Jekyll2020-01-21T22:45:43+00:00https://elipapa.github.io/feed.xmlEliseo PapaEliseo Papa's thoughts, ideas and projects. Here so they can be safely ignored.How i fixed my corrupted sparsebundle image2015-11-02T13:01:00+00:002015-11-02T13:01:00+00:00https://elipapa.github.io/blog/repair-sparsebundles<p>I save all my paper scans and important documents in an encrypted sparsebundle. In lieu of proper backup, I place the file in my dropbox. However, when I got too defiant and tried to use the same image from more than one computers connected to dropbox, the image was corrupted and could not be mounted anymore. Panic.</p>
<p>How do i repair the sparsebundle image i had in dropbox?</p>
<p>Apparently this happened more than once. A quick google search returns a bunch of questions on stack overflow trying to find a way around it. Not many with encouraging results. Sparsebundles are treated as directories by dropbox and it’s not possible to restore entire directories at once from the graphical interface. But encrypted bundles are definitely not directories and as soon as two computers change a chunk at the same time, mayhem occurs.</p>
<p>After a while, however, I did come up with something: There is a python command called <a href="https://github.com/clark800/dropbox-restore">dropbox-restore</a> which interfaces with the dropbox API and retrieves every independent file in the directory (or sparsebundle) at a certain time stamp.</p>
<p>I haven’t tested it, but I ended up installing installing an alternative called <a href="https://www.revisionsapp.com">revisions app</a>, which gives a graphical interface to the same functionality and a lot more ways of interacting with the dropbox API. It was free to install and after 2-3 hours of indexing work, i was able to succesfully retrieve my encrypted sparsebundle down to a date when it was not corrupted.</p>
<p>This near disaster taught me the importance of <a href="http://lifehacker.com/psa-dropbox-shouldnt-be-your-sole-backup-for-your-file-1612803794">splitting syncing and backup</a>.</p>elipapaI save all my paper scans and important documents in an encrypted sparsebundle. In lieu of proper backup, I place the file in my dropbox. However, when I got too defiant and tried to use the same image from more than one computers connected to dropbox, the image was corrupted and could not be mounted anymore. Panic.You Are What You Eat - an interactive visualization of food data2015-06-02T15:45:00+00:002015-06-02T15:45:00+00:00https://elipapa.github.io/blog/dcjs_to_visualize_food_diaries<p>I have created a <a href="http://elipapa.github.io/youarewhatyoueat/">visual dashboard</a>, which allows to explore daily food diaries. I wanted to try using some of the interactive capabilities of <a href="https://d3js.org/">D3.js</a> and <a href="http://square.github.io/crossfilter/">Crossfilter</a>. I have ended up leveraging the <a href="https://dc-js.github.io/dc.js/">dc.js</a> library, which integrates the two with good templates.</p>
<p>The data is scraped from 200 users of the MyFitnessPal app, who shared their food logs publicly. I used <a href="http://um.ai">um.ai</a> ontology of food to categorize each food item in its food group and to estimate the calories and macronutrients consumed.</p>
<p>While quite easy to use and visually appealing, using dc.js still required a bit of attention. Sure, my javascript skills are minimal, but there were a number of little details which required to dig into stackoverflow,etc.
It’s a good tool to create and expose stable dashboards, but probably not great for quick prototyping of different metrics.</p>elipapaI have created a visual dashboard, which allows to explore daily food diaries. I wanted to try using some of the interactive capabilities of D3.js and Crossfilter. I have ended up leveraging the dc.js library, which integrates the two with good templates.SLiMEbook2015-06-01T10:00:00+00:002015-06-01T10:00:00+00:00https://elipapa.github.io/blog/slimebook<p>I published an IBD paper <a href="http://journals.plos.org/plosone/article?id=10.1371/journal.pone.0039242">in PLoSONE</a> - showing one of the first application of ML methods to microbiome data. We dubbed the code used for the analysis SLiME.</p>
<p>The SLiME code (written in <a href="https://www.r-project.org/">R</a>) became outdated pretty quickly. We decided that the code, as it was packaged for the publication of the paper, should not be in the open anymore.</p>
<p>In its place, <a href="http://nbviewer.ipython.org/github/elipapa/SLiMEbook/blob/master/SLiMEbook.ipynb">this ipython notebook</a> will serve as a repository of the analysis. I hope this can be the starting point for others trying to follow the same approach and improve upon it. It leverages python, sklearn and pandas.</p>
<p>Check the full <a href="https://github.com/elipapa/SLiMEbook/">repo</a>, which also contains the <a href="https://github.com/elipapa/SLiMEbook/tree/master/data/chimp">raw data</a></p>elipapaI published an IBD paper in PLoSONE - showing one of the first application of ML methods to microbiome data. We dubbed the code used for the analysis SLiME.What is the fastest way to find duplicate pictures?2015-02-19T00:00:00+00:002015-02-19T00:00:00+00:00https://elipapa.github.io/blog/fastest-way-to-find-dups<p>I needed to clean up duplicates photos from my personal library. And because I could not choose which duplicate finder to try, I have decided to test them all. Amongst the free tools that were correct in identifying all duplicates in my test, <a href="https://github.com/jvirkki/dupd">dupd</a> was the fastest. Other <a href="https://www.python.org/">python</a> and <a href="http://www.perl.org/">perl</a> based solution did also very well, often better than their C/C++ colleagues.</p>
<h2 id="why-a-duplicate-file-finder">Why a duplicate file finder?</h2>
<p>As a parent, my picture collection has mushroomed in size. I do try to save everything in one place and have backups, but it is difficult to keep track of it. Too afraid of losing family memories, often I end up downloading pictures from my phone multiple times, “just in case”. Now that I have reached the limits of my hard drive space, I figured the best thing to do was to remove some duplicate pictures.</p>
<p>One could try with iPhoto or some other graphical interface, but the approach is simply too slow for a large library (over 100GB). The obvious choice was to search for a command line tool.</p>
<p>I quickly realized that <strong>there are far too many tools</strong> which have been written for this task. And no easy way to find which one is best. So I decided to compare the speed of most of them.</p>
<h2 id="a-speed-comparison">A speed comparison</h2>
<p>Wikipedia has a pretty complete <a href="http://en.wikipedia.org/wiki/List_of_duplicate_file_finders">list of duplicate file finders</a>. From there I downloaded and installed all free/open source command line tools.</p>
<p>I have excluded from this comparison those tools which could not be readily installed on my macbook (eg. Makefile would have needed fixing) or appear to have very limited support (less than 10 stars on github, slow or stagnant development, etc.).
I did not take into considerations those tools that do not have the option of a “dry run” or simple listing of duplicates, but instead attempt to delete or hardlink the duplicate files. I find this behaviour too aggressive for most users. Definitely too risky to run on the folder containing the pictures of my kids.</p>
<p>Finally, I excluded those files that failed to find all correct duplicates in my test folder. The folder was designed to contain 1195 duplicates in 325 clusters. Here are the results, tested on a 7GB mixture of pictures, videos, symlinks, small files and recursively nested files:</p>
<iframe width="700" height="450" seamless="" frameborder="0" scrolling="no" src="https://docs.google.com/spreadsheets/d/1zEaULNBhh-ynwpDnLDZT56rAOeupGKgoSblyREEfSbw/pubchart?oid=377382474&format=interactive"></iframe>
<p>Since I keep my pictures in a separate NAS storage, it is also useful how much each of this methods hinges on memory or CPU:</p>
<iframe width="700" height="450" seamless="" frameborder="0" scrolling="no" src="https://docs.google.com/spreadsheets/d/1zEaULNBhh-ynwpDnLDZT56rAOeupGKgoSblyREEfSbw/pubchart?oid=980882643&format=interactive"></iframe>
<iframe width="700" height="450" seamless="" frameborder="0" scrolling="no" src="https://docs.google.com/spreadsheets/d/1zEaULNBhh-ynwpDnLDZT56rAOeupGKgoSblyREEfSbw/pubchart?oid=903055661&format=interactive"></iframe>
<h2 id="conclusion">Conclusion</h2>
<p><a href="https://github.com/jvirkki/dupd">Dupd</a> was the clear speed winner, also with an acceptable memory footprint. <a href="http://code.google.com/p/liten/">liten</a> and <a href="http://ssokolow.com/scripts/#fastdupes.py">fastdupes</a> come close second and may be slightly more portable as they do not require to be compiled. Compiling of the C/C++ tools tend to be a little fragile out of the main UNIX distros, which is a problem when working on a NAS. <strong>I ended up using <a href="http://ssokolow.com/scripts/#fastdupes.py">fastdupes</a></strong>.</p>
<p>It is interesting to see how the (arguably) best known solution <a href="https://code.google.com/p/fdupes/">fdupes</a> was also the slowest. Though it remains one of the only tools which can do byte-by-byte comparison. Both the fastest and second fastest tool rely on <a href="http://www.sqlite.org/">SQLite databases</a> and allow you to explore duplicates interactively, after they run.</p>
<p>Please let me know if I forgot any other tool which should have been in this list. The commands included in the analysis were:</p>
<ul>
<li><a href="http://duff.dreda.org/">duff</a> (C)</li>
<li><a href="http://www.beautylabs.net/software/dupseek.html">dupseek</a> (Perl)</li>
<li><a href="http://ssokolow.com/scripts/#fastdupes.py">fastdupes.py</a> (Python)</li>
<li><a href="http://www.iredale.net/p/by-permalink/449447C8-EACB-11DC-B77B-21EB6225452B/">fdf</a> (perl)</li>
<li><a href="http://freshmeat.net/projects/fdupe/">fdupe</a> (perl)</li>
<li><a href="https://github.com/adrianlopezroche/fdupes">fdupes</a>(C)</li>
<li><a href="https://github.com/jbruchon/fdupes-jody">fdupes-jody</a> a speedier fork of fdupes (C)</li>
<li><a href="http://code.google.com/p/liten/">liten</a> (Python)</li>
<li><a href="http://code.google.com/p/liten2/">liten2</a> (Python)</li>
<li><a href="http://rdfind.pauldreik.se/">rdfind</a> (C)</li>
<li><a href="http://ssdeep.sourceforge.net/">ssdeep</a> (C) also does partial matches</li>
<li><a href="http://code.activestate.com/recipes/551777/">python active state recipe</a> (Python)</li>
</ul>I needed to clean up duplicates photos from my personal library. And because I could not choose which duplicate finder to try, I have decided to test them all. Amongst the free tools that were correct in identifying all duplicates in my test, dupd was the fastest. Other python and perl based solution did also very well, often better than their C/C++ colleagues.fix ruby and bundler after upgrading openssl2014-12-19T13:01:00+00:002014-12-19T13:01:00+00:00https://elipapa.github.io/blog/ruby_openssl_eerror<p>I had trouble running <a href="http://bundler.io">bundler</a> on this blog because of a ruby error. Though I am not sure what I have broken for this to occur, it appears that ruby could not find openssl libraries anymore. A quick search on <a href="http://stackoverflow.com/questions/25492787/ruby-bundle-symbol-not-found-sslv2-client-method-loaderror">stack overflow</a> pointed me on the <a href="https://github.com/sstephenson/rbenv/issues/610">right direction</a>. Apparently upgrading openssl via homebrew breaks ruby dependencies. What I had to do was to install another ruby version with my ruby manager of choice (<a href="https://github.com/sstephenson/rbenv">rbenv</a>):</p>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>CFLAGS='-g -O2' RUBY_CONFIGURE_OPTS=--with-openssl-dir=`brew --prefix openssl` rbenv install 2.2.0
rbenv global 2.2.02
rbenv rehash
gem install bundler
</code></pre></div></div>
<p>And that was enough to bring me back to square one.</p>I had trouble running bundler on this blog because of a ruby error. Though I am not sure what I have broken for this to occur, it appears that ruby could not find openssl libraries anymore. A quick search on stack overflow pointed me on the right direction. Apparently upgrading openssl via homebrew breaks ruby dependencies. What I had to do was to install another ruby version with my ruby manager of choice (rbenv):Why I switched to markdown for my CV2012-09-20T15:46:00+00:002012-09-20T15:46:00+00:00https://elipapa.github.io/blog/why-i-switched-to-markdown-for-my-cv<p>I have previously used <a href="http://www.latex-project.org/">LaTeX</a> to typeset my <em>curriculum vitae</em>, as it invariably produces a beautiful looking document. I have now become frustrated at how long it takes me to relearn LaTeX from scratch every time I want to change something. Plus my <code class="language-plaintext highlighter-rouge">.tex</code> source file was becoming completely unreadable. In an attempt to clear up the mess, I decided to put all the content in a markdown file and use CSS to style it. The result is <a href="https://elipapa.github.io/markdown-cv/">markdown-cv</a>, which can be forked and used as a template by anyone who wants to do the same thing.</p>
<p>I have kept my <em>curriculum vitae</em> in <a href="http://www.latex-project.org/">LaTeX</a> for a long time. My workflow included a plain text <a href="http://www.gnu.org/software/emacs/">editor</a> and a <a href="http://www.tug.org/mactex/">Mac installation of LaTeX</a>. LaTeX’s elegant typography meant my CV would always look a tad better than its corresponding version in word, and I stuck to it. I also enjoyed having to deal only with plain text files, saving me from the converting files from one word processor to the other. Not to mention that having a single file to update, rather than a series of word processor files scattered everywhere, kept me more disciplined about updating it regularly.</p>
<p>However it was definitely not perfect. For a document as short as my CV, LaTeX was probably overkill. One or two hours spent in word can easily produce a new style or version of the CV, while doing the same thing in LaTeX would always require much more tinkering.
I began to get tired of sparkling small LaTeX commands throughout the file, adding <code class="language-plaintext highlighter-rouge">\vskip</code> and page breaks to maintain the overall appearance. Albeit plain text, <em>the LaTeX file which contained my CV became increasingly less readable.</em> Though the primary purpose of LaTeX should be to separate content from presentation, the division was becoming to me always less evident. The last drop was when I tried to update the overall look of my resume. I ended up having to rearrange paragraphs and dates, while making up obscure LaTeX macros just to move the years to a different place or change the color of a particular element.</p>
<p>Having recently worked with static blog engines, I immediately thought the division between content and presentation was much more elegant there. The content is usually kept in <a href="http://daringfireball.net/projects/markdown/">Markdown</a>, which is very easily readable in any text editor and can be readily converted to HTML. Styling the content is done using CSS and can be done differently for print media or for visualization in a web browser using <a href="http://www.alistapart.com/articles/goingtoprint/">media queries</a>.</p>
<p>So I proceded to transfer my CV in Markdown format, created a simple HTML template and rendered it using <a href="https://github.com/mojombo/jekyll">jekyll</a>, the engine used by <a href="http://www.github.com">GitHub</a> to render static pages (and what I use to render this blog). I created a CSS to render the page in a style inspired by <a href="http://kjhealy.github.com/kjh-vita/">kjhealy’s vita template</a>.</p>
<p>To obtain a <a href="https://elipapa.github.io/downloads/cv.pdf">pdf</a> version of the cv, I am using the great <a href="http://code.google.com/p/wkhtmltopdf/">wkhtmltopdf</a>, which I call by means of a <a href="http://rake.rubyforge.org/">rake</a> task: <code class="language-plaintext highlighter-rouge">rake pdf</code>.</p>
<p>You can see the various pieces of the workflow arranged on <a href="https://github.com/elipapa/markdown-cv">GitHub</a> (Hosting on GitHub there also means I can maintain different versions of the same file, eg. one resume for tech jobs and another for academic purposes). I am quite happy with the <a href="https://elipapa.github.io/downloads/cv.pdf">final result</a>.</p>
<p>I can’t say that this workflow was completely succesful in doing away with the tinkering, but at the very least the main content of my curriculum vitae is now available in plain readable text. I can easily edit it without having to remember any LaTeX commands. That and I needed the CSS practice…</p>elipapaI have previously used LaTeX to typeset my curriculum vitae, as it invariably produces a beautiful looking document. I have now become frustrated at how long it takes me to relearn LaTeX from scratch every time I want to change something. Plus my .tex source file was becoming completely unreadable. In an attempt to clear up the mess, I decided to put all the content in a markdown file and use CSS to style it. The result is markdown-cv, which can be forked and used as a template by anyone who wants to do the same thing.Ph.D Defense talk2012-05-16T15:47:00+00:002012-05-16T15:47:00+00:00https://elipapa.github.io/blog/phd-defense-talk<p>I’ve seen many share slides on <a href="http://www.slideshare.net">slideshare</a> before, but I was impressed with the ease and the clean interface of <a href="https://speakerdeck.com/">Speaker Deck</a>. To test it out, I have uploaded my <a href="https://speakerdeck.com/elipapa/phd-defense-talk-1">Ph.D defense talk</a>.</p>
<p>Not only the permalink is quite intuitive, but one can also embed the presentation in a page quite easily:</p>
<script async="" class="speakerdeck-embed" data-id="4fb3bab625741e001f029e0a" data-ratio="1.3333333333333333" src="//speakerdeck.com/assets/embed.js"></script>
<p>Needless to say I am impressed. This may be a good way to store presentations: arrive in the lecture hall, use whatever computer is connected to the projector and browse to <a href="https://speakerdeck.com/">Speaker Deck</a>. Click fullscreen and go. No more thumb drives or lost video connectors.</p>elipapaI’ve seen many share slides on slideshare before, but I was impressed with the ease and the clean interface of Speaker Deck. To test it out, I have uploaded my Ph.D defense talk.Poster for the host-microbiome interactions conference at the Sanger Institute2012-05-08T22:33:00+00:002012-05-08T22:33:00+00:00https://elipapa.github.io/blog/poster-host-microbiome-interactions-conf<p>My abstract was selected for the <a href="https://registration.hinxton.wellcome.ac.uk/display_info.asp?id=271">Host-Microbiome Interactions in Health and Disease</a> conference at the Sanger Institute, which is great news. Already a few hours of the meeting went by and I don’t think I ever talked so much microbiome in so little time. Here is a <a href="https://elipapa.github.io/downloads/documents/ibdML-poster.pdf">low resolution pdf of my poster</a>.</p>
<p><img src="https://elipapa.github.io/assets/images/ibdML-poster.png" alt="poster preview" width="300" style="align:center;" /></p>
<p>I would probably design it differently, now that I have had the chance to see it on the wall. Too much text and the font was too tiny.</p>elipapaMy abstract was selected for the Host-Microbiome Interactions in Health and Disease conference at the Sanger Institute, which is great news. Already a few hours of the meeting went by and I don’t think I ever talked so much microbiome in so little time. Here is a low resolution pdf of my poster.