{"id":13619,"date":"2016-12-02T15:52:52","date_gmt":"2016-12-02T14:52:52","guid":{"rendered":"http:\/\/codisec.com\/?p=13619"},"modified":"2023-03-22T16:29:59","modified_gmt":"2023-03-22T15:29:59","slug":"binary-data-visualization","status":"publish","type":"post","link":"https:\/\/codisec.com\/binary-data-visualization\/","title":{"rendered":"Binary data visualization"},"content":{"rendered":"<blockquote><p>This page contains examples of Veles visualization. For explanation on how they actually work check out <a href=\"https:\/\/codisec.com\/binary-visualization-explained\/\" target=\"_blank\" rel=\"noopener\">Binary&nbsp;visualization&nbsp;explained<\/a>.<\/p><\/blockquote>\n<p>By testing Veles visualizations on numerous files we found that different types of data look very differently.<br \/>\nWe can easily notice the differences between a bitmap, a mobi file, a java .class file and an x86 compiled binary.<\/p>\n<div class=\"row\">\n<div class=\"col-sm-6\">\n<div style=\"overflow: hidden;\">\n<div id=\"carousel-example-generic2\" class=\"carousel slide\" data-ride=\"carousel\">\n<div class=\"carousel-inner\">\n<div class=\"active item\">\n<div class=\"modal-carousel-hybrid-source-image-wrapper modal-carousel-hybrid-source-image-wrapper-1\"><img class=\"modal-carousel-hybrid-source-image\" src=\"\/wp-content\/uploads\/2016\/12\/binary-data-visualization-1.png\" alt=\"...\" data-target=\"#myModal\" data-toggle=\"modal\" \/><\/div>\n<\/div>\n<\/div>\n<\/div>\n<div id=\"myModal\" class=\"fade modal\" tabindex=\"-1\">\n<div class=\"modal-dialog\">\n<div class=\"modal-content\">\n<button type=\"button\" class=\"close\" data-dismiss=\"modal\" aria-label=\"Close\"><span aria-hidden=\"true\">&times;<\/span><\/button><\/p>\n<div id=\"modal-carousel-hybrid-popup\" class=\"modal-body\"><img src=\"#\" \/><\/div>\n<\/div>\n<\/div>\n<\/div>\n<\/div>\n<\/div>\n<div class=\"col-sm-6\">\n<div style=\"overflow: hidden;\">\n<div id=\"carousel-example-generic2\" class=\"carousel slide\" data-ride=\"carousel\">\n<div class=\"carousel-inner\">\n<div class=\"active item\">\n<div class=\"modal-carousel-hybrid-source-image-wrapper modal-carousel-hybrid-source-image-wrapper-2\"><img class=\"modal-carousel-hybrid-source-image\" src=\"\/wp-content\/uploads\/2016\/12\/binary-data-visualization-4.png\" alt=\"...\" data-target=\"#myModal\" data-toggle=\"modal\" \/><\/div>\n<\/div>\n<\/div>\n<\/div>\n<div id=\"myModal\" class=\"fade modal\" tabindex=\"-1\">\n<div class=\"modal-dialog\">\n<div class=\"modal-content\">\n<div id=\"modal-carousel-hybrid-popup\" class=\"modal-body\"><img src=\"#\" \/><\/div>\n<\/div>\n<\/div>\n<\/div>\n<\/div>\n<\/div>\n<div class=\"col-sm-6\">\n<div style=\"overflow: hidden;\">\n<div id=\"carousel-example-generic2\" class=\"carousel slide\" data-ride=\"carousel\">\n<div class=\"carousel-inner\">\n<div class=\"active item\">\n<div class=\"modal-carousel-hybrid-source-image-wrapper modal-carousel-hybrid-source-image-wrapper-3\"><img class=\"modal-carousel-hybrid-source-image\" src=\"\/wp-content\/uploads\/2016\/12\/binary-data-visualization-3.png\" alt=\"...\" data-target=\"#myModal\" data-toggle=\"modal\" \/><\/div>\n<\/div>\n<\/div>\n<\/div>\n<div id=\"myModal\" class=\"fade modal\" tabindex=\"-1\">\n<div class=\"modal-dialog\">\n<div class=\"modal-content\">\n<div id=\"modal-carousel-hybrid-popup\" class=\"modal-body\"><img src=\"#\" \/><\/div>\n<\/div>\n<\/div>\n<\/div>\n<\/div>\n<\/div>\n<div class=\"col-sm-6\">\n<div style=\"overflow: hidden;\">\n<div id=\"carousel-example-generic2\" class=\"carousel slide\" data-ride=\"carousel\">\n<div class=\"carousel-inner\">\n<div class=\"active item\">\n<div class=\"modal-carousel-hybrid-source-image-wrapper modal-carousel-hybrid-source-image-wrapper-4\"><img class=\"modal-carousel-hybrid-source-image\" src=\"\/wp-content\/uploads\/2016\/12\/binary-data-visualization-2.png\" alt=\"...\" data-target=\"#myModal\" data-toggle=\"modal\" \/><\/div>\n<\/div>\n<\/div>\n<\/div>\n<div id=\"myModal\" class=\"fade modal\" tabindex=\"-1\">\n<div class=\"modal-dialog\">\n<div class=\"modal-content\">\n<div id=\"modal-carousel-hybrid-popup\" class=\"modal-body\"><img src=\"#\" \/><\/div>\n<\/div>\n<\/div>\n<\/div>\n<\/div>\n<\/div>\n<\/div>\n<hr \/>\n<p>On a side note &#8211; visualizations of compressed or encrypted data look like a bunch of noise. Any trace of pattern in&nbsp;the&nbsp;visualized data immediately stands. For example, compare the gpg encrypted data and a .zip archive below. visualization makes it easy to spot headers in a zip archive. By switching to a &#8220;layered digram&#8221; mode of visualization we&nbsp;can immediately locate the headers at the end of the file. And not only that \u2013 we can also recognize certain patterns in&nbsp;the compressed stream (like the line on the right side of image).<\/p>\n<div class=\"row\">\n<div class=\"col-sm-6\">\n<div style=\"overflow: hidden;\">\n<div id=\"carousel-example-generic2\" class=\"carousel slide\" data-ride=\"carousel\">\n<div class=\"carousel-inner\">\n<div class=\"active item\">\n<div class=\"modal-carousel-hybrid-source-image-wrapper modal-carousel-hybrid-source-image-wrapper-5\"><img class=\"modal-carousel-hybrid-source-image\" src=\"\/wp-content\/uploads\/2016\/12\/binary-data-visualization-7.png\" alt=\"...\" data-target=\"#myModal\" data-toggle=\"modal\" \/><\/div>\n<\/div>\n<\/div>\n<\/div>\n<div id=\"myModal\" class=\"fade modal\" tabindex=\"-1\">\n<div class=\"modal-dialog\">\n<div class=\"modal-content\">\n<div id=\"modal-carousel-hybrid-popup\" class=\"modal-body\"><img src=\"#\" \/><\/div>\n<\/div>\n<\/div>\n<\/div>\n<\/div>\n<\/div>\n<div class=\"col-sm-6\">\n<div style=\"overflow: hidden;\">\n<div id=\"carousel-example-generic2\" class=\"carousel slide\" data-ride=\"carousel\">\n<div class=\"carousel-inner\">\n<div class=\"active item\">\n<div class=\"modal-carousel-hybrid-source-image-wrapper modal-carousel-hybrid-source-image-wrapper-6\"><img class=\"modal-carousel-hybrid-source-image\" src=\"\/wp-content\/uploads\/2016\/12\/binary-data-visualization-5.png\" alt=\"...\" data-target=\"#myModal\" data-toggle=\"modal\" \/><\/div>\n<\/div>\n<\/div>\n<\/div>\n<div id=\"myModal\" class=\"fade modal\" tabindex=\"-1\">\n<div class=\"modal-dialog\">\n<div class=\"modal-content\">\n<div id=\"modal-carousel-hybrid-popup\" class=\"modal-body\"><img src=\"#\" \/><\/div>\n<\/div>\n<\/div>\n<\/div>\n<\/div>\n<\/div>\n<div class=\"col-sm-6\">\n<div style=\"overflow: hidden;\">\n<div id=\"carousel-example-generic2\" class=\"carousel slide\" data-ride=\"carousel\">\n<div class=\"carousel-inner\">\n<div class=\"active item\">\n<div class=\"modal-carousel-hybrid-source-image-wrapper modal-carousel-hybrid-source-image-wrapper-7\"><img class=\"modal-carousel-hybrid-source-image\" src=\"\/wp-content\/uploads\/2016\/12\/binary-data-visualization-6.png\" alt=\"...\" data-target=\"#myModal\" data-toggle=\"modal\" \/><\/div>\n<\/div>\n<\/div>\n<\/div>\n<div id=\"myModal\" class=\"fade modal\" tabindex=\"-1\">\n<div class=\"modal-dialog\">\n<div class=\"modal-content\">\n<div id=\"modal-carousel-hybrid-popup\" class=\"modal-body\"><img src=\"#\" \/><\/div>\n<\/div>\n<\/div>\n<\/div>\n<\/div>\n<\/div>\n<\/div>\n<hr \/>\n<p>Back to compiled binaries. We found out that any machine code looks roughly similar, but different architectures have their characteristic traits that can be used to recognize them. Below is the same binary compiled using three different architectures:<\/p>\n<div class=\"row\">\n<div class=\"col-sm-6\">\n<div style=\"overflow: hidden;\">\n<div id=\"carousel-example-generic2\" class=\"carousel slide\" data-ride=\"carousel\">\n<div class=\"carousel-inner\">\n<div class=\"active item\">\n<div class=\"modal-carousel-hybrid-source-image-wrapper modal-carousel-hybrid-source-image-wrapper-9\"><img class=\"modal-carousel-hybrid-source-image\" src=\"\/wp-content\/uploads\/2016\/12\/binary-data-visualization-9.png\" alt=\"...\" data-target=\"#myModal\" data-toggle=\"modal\" \/><\/div>\n<\/div>\n<\/div>\n<\/div>\n<div id=\"myModal\" class=\"fade modal\" tabindex=\"-1\">\n<div class=\"modal-dialog\">\n<div class=\"modal-content\">\n<div id=\"modal-carousel-hybrid-popup\" class=\"modal-body\"><img src=\"#\" \/><\/div>\n<\/div>\n<\/div>\n<\/div>\n<\/div>\n<\/div>\n<div class=\"col-sm-6\">\n<div style=\"overflow: hidden;\">\n<div id=\"carousel-example-generic2\" class=\"carousel slide\" data-ride=\"carousel\">\n<div class=\"carousel-inner\">\n<div class=\"active item\">\n<div class=\"modal-carousel-hybrid-source-image-wrapper modal-carousel-hybrid-source-image-wrapper-10\"><img class=\"modal-carousel-hybrid-source-image\" src=\"\/wp-content\/uploads\/2016\/12\/binary-data-visualization-10.png\" alt=\"...\" data-target=\"#myModal\" data-toggle=\"modal\" \/><\/div>\n<\/div>\n<\/div>\n<\/div>\n<div id=\"myModal\" class=\"fade modal\" tabindex=\"-1\">\n<div class=\"modal-dialog\">\n<div class=\"modal-content\">\n<div id=\"modal-carousel-hybrid-popup\" class=\"modal-body\"><img src=\"#\" \/><\/div>\n<\/div>\n<\/div>\n<\/div>\n<\/div>\n<\/div>\n<div class=\"col-sm-6\">\n<div style=\"overflow: hidden;\">\n<div id=\"carousel-example-generic2\" class=\"carousel slide\" data-ride=\"carousel\">\n<div class=\"carousel-inner\">\n<div class=\"active item\">\n<div class=\"modal-carousel-hybrid-source-image-wrapper modal-carousel-hybrid-source-image-wrapper-11\"><img class=\"modal-carousel-hybrid-source-image\" src=\"\/wp-content\/uploads\/2016\/12\/binary-data-visualization-11.png\" alt=\"...\" data-target=\"#myModal\" data-toggle=\"modal\" \/><\/div>\n<\/div>\n<\/div>\n<\/div>\n<div id=\"myModal\" class=\"fade modal\" tabindex=\"-1\">\n<div class=\"modal-dialog\">\n<div class=\"modal-content\">\n<div id=\"modal-carousel-hybrid-popup\" class=\"modal-body\"><img src=\"#\" \/><\/div>\n<\/div>\n<\/div>\n<\/div>\n<\/div>\n<\/div>\n<\/div>\n<hr \/>\n<p>Ok, now let&#8217;s take a look at a specific file. For this demo, we&#8217;ll use libc.so from ubuntu 14.04.<\/p>\n<pre class=\"nums:false wrap:true\">&gt; file libc-2.22.so\r\nlibc-2.22.so: ELF 64-bit LSB shared object, x86-64, version 1 (GNU\/Linux), dynamically linked (uses shared libs), for GNU\/Linux 2.6.32, not stripped<\/pre>\n<p><iframe loading=\"lazy\" style=\"margin-top:15px;\" width=\"560\" height=\"315\" src=\"https:\/\/www.youtube.com\/embed\/UworRSD7iEk\" frameborder=\"0\" allowfullscreen><\/iframe><\/p>\n<h2 style=\"margin-top: 45px;\">Recognising x86-64 architecture<\/h2>\n<p>As mentioned in the video, we can recognise x86-64 code by finding 2 characteristic bars in the trigram view. Such a pair of bars means that there is a common sequence of 2 similar bytes (let&#8217;s call them x and y). One of the bars will correspond to trigrams &lt;something, x, y&gt;, while the second bar will be made of trigrams &lt;x, y, something&gt;.<\/p>\n<p>So why do we see these bars in x86-64 machine code? It turns out that the default operand size for many instructions in&nbsp;64-bit mode is 32 bits. If we want to use a 64-bit register we need to add a REX prefix. That means we prefix the&nbsp;instruction opcode with an additional byte with a value 0x40 + flags on lower 4 bits. In particular, many variants of&nbsp;MOV instruction will have a 2 byte opcode 0x4X 0x8Y (where X and Y depend on exactly which version of the instruction we used). Since MOV is an extremely common instruction, there will be a lot of those digrams in any x86 64-bit binary and we will clearly see the bars in trigram view. It&#8217;s also worth mentioning that another common instruction &#8211; LEA &#8211; happens to have 0x4X 0x8D opcode, which makes the bars even more visible.<\/p>\n<h2>.gnu.hash<\/h2>\n<p>As mentioned in the video the .gnu.hash section of ELF is made of 3 distinct parts:<\/p>\n<ul>\n<li>\n<h4><a href=\"https:\/\/en.wikipedia.org\/wiki\/Bloom_filter\" target=\"_blank\" rel=\"noopener\">Bloom filter <i class=\"fa fa-link\"><\/i><\/a><\/h4>\n<p>This is a probabilistic set with a possibility of false positives. Basically, when inserting an object it&#8217;s hashed into a few bit locations and all those bits are set to 1. To check if the object is already in the set we can just check all of those bits &#8211; if&nbsp;at&nbsp;least one of them has a value of 0 then we know the object is not in the set. The idea behind including this in the ELF is to avoid (if possible) the more expensive hashmap lookups.<\/p>\n<p>In the visualization we can see that the Bloom filter is mostly random, but there are significantly more 0 bits than 1 bits in it. There are more numbers near the (0, 0, 0) corner of the cube. Additionally, we can see clusters of points near positions corresponding to values with just one or two 1s in binary representation.<\/li>\n<li>\n<h4>Hash buckets<\/h4>\n<p>Each value is the lowest index in the dynamic symbol table corresponding to a given hash. What we see in the visualization is relatively small values represented on 32-bit integers.<\/li>\n<li>\n<h4>Hash values<\/h4>\n<p>These are computed hash values. As expected, they look like a bunch of white noise in a visualization.<\/p>\n<p>For a more detailed description of how this section works, check out <a style=\"color: #f1851a;\" href=\"https:\/\/blogs.oracle.com\/ali\/entry\/gnu_hash_elf_sections\" target=\"_blank\" rel=\"noopener\">this link<\/a><\/li>\n<\/ul>\n","protected":false},"excerpt":{"rendered":"<p>This page contains examples of Veles visualization. For explanation on how they actually work check out Binary&nbsp;visualization&nbsp;explained. By testing Veles visualizations on numerous files we found that different types of data look very differently. We can easily notice the differences<span class=\"ellipsis\">&hellip;<\/span> <a href=\"https:\/\/codisec.com\/binary-data-visualization\/\"><\/p>\n<div class=\"read-more\">Read more &#8250;<\/div>\n<p><!-- end of .read-more --><\/a><\/p>\n","protected":false},"author":5,"featured_media":0,"comment_status":"closed","ping_status":"open","sticky":false,"template":"","format":"standard","meta":[],"categories":[22,24],"tags":[],"yoast_head":"<!-- This site is optimized with the Yoast SEO plugin v16.1.1 - https:\/\/yoast.com\/wordpress\/plugins\/seo\/ -->\n<link rel=\"canonical\" href=\"https:\/\/codisec.com\/binary-data-visualization\/\" \/>\n<meta property=\"og:locale\" content=\"en_US\" \/>\n<meta property=\"og:type\" content=\"article\" \/>\n<meta property=\"og:title\" content=\"Binary data visualization - CodiSec\" \/>\n<meta property=\"og:description\" content=\"This page contains examples of Veles visualization. For explanation on how they actually work check out Binary&nbsp;visualization&nbsp;explained. By testing Veles visualizations on numerous files we found that different types of data look very differently. We can easily notice the differences&hellip; Read more &#8250;\" \/>\n<meta property=\"og:url\" content=\"https:\/\/codisec.com\/binary-data-visualization\/\" \/>\n<meta property=\"og:site_name\" content=\"CodiSec\" \/>\n<meta property=\"article:published_time\" content=\"2016-12-02T14:52:52+00:00\" \/>\n<meta property=\"article:modified_time\" content=\"2023-03-22T15:29:59+00:00\" \/>\n<meta property=\"og:image\" content=\"https:\/\/codisec.com\/wp-content\/uploads\/2016\/12\/binary-data-visualization-1.png\" \/>\n<meta name=\"twitter:card\" content=\"summary\" \/>\n<meta name=\"twitter:label1\" content=\"Est. reading time\">\n\t<meta name=\"twitter:data1\" content=\"3 minutes\">\n<script type=\"application\/ld+json\" class=\"yoast-schema-graph\">{\"@context\":\"https:\/\/schema.org\",\"@graph\":[{\"@type\":\"WebSite\",\"@id\":\"https:\/\/codisec.com\/#website\",\"url\":\"https:\/\/codisec.com\/\",\"name\":\"CodiSec\",\"description\":\"PENETRATION TESTING  AND MALWARE ANALYSIS\",\"potentialAction\":[{\"@type\":\"SearchAction\",\"target\":\"https:\/\/codisec.com\/?s={search_term_string}\",\"query-input\":\"required name=search_term_string\"}],\"inLanguage\":\"en-US\"},{\"@type\":\"ImageObject\",\"@id\":\"https:\/\/codisec.com\/binary-data-visualization\/#primaryimage\",\"inLanguage\":\"en-US\",\"url\":\"\/wp-content\/uploads\/2016\/12\/binary-data-visualization-1.png\",\"contentUrl\":\"\/wp-content\/uploads\/2016\/12\/binary-data-visualization-1.png\"},{\"@type\":\"WebPage\",\"@id\":\"https:\/\/codisec.com\/binary-data-visualization\/#webpage\",\"url\":\"https:\/\/codisec.com\/binary-data-visualization\/\",\"name\":\"Binary data visualization - CodiSec\",\"isPartOf\":{\"@id\":\"https:\/\/codisec.com\/#website\"},\"primaryImageOfPage\":{\"@id\":\"https:\/\/codisec.com\/binary-data-visualization\/#primaryimage\"},\"datePublished\":\"2016-12-02T14:52:52+00:00\",\"dateModified\":\"2023-03-22T15:29:59+00:00\",\"author\":{\"@id\":\"https:\/\/codisec.com\/#\/schema\/person\/76ede82820a90a251ac07a0dc4f9fd87\"},\"breadcrumb\":{\"@id\":\"https:\/\/codisec.com\/binary-data-visualization\/#breadcrumb\"},\"inLanguage\":\"en-US\",\"potentialAction\":[{\"@type\":\"ReadAction\",\"target\":[\"https:\/\/codisec.com\/binary-data-visualization\/\"]}]},{\"@type\":\"BreadcrumbList\",\"@id\":\"https:\/\/codisec.com\/binary-data-visualization\/#breadcrumb\",\"itemListElement\":[{\"@type\":\"ListItem\",\"position\":1,\"item\":{\"@type\":\"WebPage\",\"@id\":\"https:\/\/codisec.com\/\",\"url\":\"https:\/\/codisec.com\/\",\"name\":\"Home\"}},{\"@type\":\"ListItem\",\"position\":2,\"item\":{\"@type\":\"WebPage\",\"@id\":\"https:\/\/codisec.com\/binary-data-visualization\/\",\"url\":\"https:\/\/codisec.com\/binary-data-visualization\/\",\"name\":\"Binary data visualization\"}}]},{\"@type\":\"Person\",\"@id\":\"https:\/\/codisec.com\/#\/schema\/person\/76ede82820a90a251ac07a0dc4f9fd87\",\"name\":\"Maciej Pytel\",\"image\":{\"@type\":\"ImageObject\",\"@id\":\"https:\/\/codisec.com\/#personlogo\",\"inLanguage\":\"en-US\",\"url\":\"https:\/\/secure.gravatar.com\/avatar\/fe6f19b992c8c1d2877f0eee7a7302ec?s=96&d=mm&r=g\",\"contentUrl\":\"https:\/\/secure.gravatar.com\/avatar\/fe6f19b992c8c1d2877f0eee7a7302ec?s=96&d=mm&r=g\",\"caption\":\"Maciej Pytel\"}}]}<\/script>\n<!-- \/ Yoast SEO plugin. -->","_links":{"self":[{"href":"https:\/\/codisec.com\/wp-json\/wp\/v2\/posts\/13619"}],"collection":[{"href":"https:\/\/codisec.com\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/codisec.com\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/codisec.com\/wp-json\/wp\/v2\/users\/5"}],"replies":[{"embeddable":true,"href":"https:\/\/codisec.com\/wp-json\/wp\/v2\/comments?post=13619"}],"version-history":[{"count":42,"href":"https:\/\/codisec.com\/wp-json\/wp\/v2\/posts\/13619\/revisions"}],"predecessor-version":[{"id":14997,"href":"https:\/\/codisec.com\/wp-json\/wp\/v2\/posts\/13619\/revisions\/14997"}],"wp:attachment":[{"href":"https:\/\/codisec.com\/wp-json\/wp\/v2\/media?parent=13619"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/codisec.com\/wp-json\/wp\/v2\/categories?post=13619"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/codisec.com\/wp-json\/wp\/v2\/tags?post=13619"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}