{"id":13760,"date":"2017-01-13T18:48:31","date_gmt":"2017-01-13T17:48:31","guid":{"rendered":"https:\/\/codisec.com\/?p=13760"},"modified":"2023-03-22T16:29:58","modified_gmt":"2023-03-22T15:29:58","slug":"binary-visualization-explained","status":"publish","type":"post","link":"https:\/\/codisec.com\/binary-visualization-explained\/","title":{"rendered":"Binary visualization explained"},"content":{"rendered":"<p>Veles visualizations are purely statistical representations of binary data. We take a sequence of bytes and visualize correlations between certain values. It doesn&#8217;t matter if it&#8217;s an executable file, a picture or a disk image &#8211; from the perspective of visualization any file is just a sequence of bytes.<\/p>\n<p>There are a few different visualization modes supported by Veles: digram, layered digram and trigram. Let&#8217;s explain them one by one.<\/p>\n<h2 style=\"margin-top: 45px;\">Digram<\/h2>\n<p>In digram visualization we look at all 2\u2011byte sequences (digrams) and compare their relative frequency in the file. We treat each 2\u2011byte sequence as pair of coordinates that we draw on a 2d surface. Let&#8217;s imagine a tiny example file made of following bytes:<\/p>\n<pre class=\"nums:false wrap:true\">02 03 05 01\r\n<\/pre>\n<p>To create a digram visualization we iterate through the file and list all the 2\u2011byte sequences we encounter: &lt;2, 3&gt;, &lt;3, 5&gt;, &lt;5, 1&gt;. We treat each pair as a 2d coordinate of a point we put in our visualization. The result is shown on the diagram below:<\/p>\n<div align=\"center\"><img class=\"alignnone wp-image-13768\" src=\"https:\/\/codisec.com\/wp-content\/uploads\/2017\/01\/digram_example.png\" alt=\"\" width=\"400\" \/><\/div>\n<p>Note that each byte (except the first and the last one in the file) is used twice, once as a coordinate on x\u2011axis and once on y\u2011axis.<\/p>\n<p>Of course real files are much larger and contain many digrams. In Veles the brightness of each point is determined by the relative frequency of each digram in the file. The most common ones will be white, while those encountered only a few times will be very dim, almost completely black. Let&#8217;s take a look at a few examples:<\/p>\n<style>\n.col-sm-6 {\n  padding-bottom: 30px;\n}\n.screenshot1:after {\n  content: \"linux shared library\";\n}\n.screenshot2:after {\n  content: \"uncompressed audio\";\n}\n.screenshot3:after {\n  content: \"text file\";\n}\n<\/style>\n<div class=\"row\">\n<div class=\"col-sm-6\">\n<div style=\"overflow: hidden;\">\n<div id=\"carousel-example-generic2\" class=\"carousel slide\" data-ride=\"carousel\">\n<div class=\"carousel-inner\">\n<div class=\"active item\">\n<div class=\"modal-carousel-hybrid-source-image-wrapper screenshot1\">\n<img loading=\"lazy\" class=\"modal-carousel-hybrid-source-image\" src=\"\/wp-content\/uploads\/2017\/01\/elf_digram.png\" alt=\"...\" width=\"1443\" height=\"983\" data-target=\"#myModal\" data-toggle=\"modal\" \/>\n<\/div>\n<\/div>\n<\/div>\n<\/div>\n<div id=\"myModal\" class=\"fade modal\" tabindex=\"-1\">\n<div class=\"modal-dialog\">\n<div class=\"modal-content\">\n<div id=\"modal-carousel-hybrid-popup\" class=\"modal-body\"><img src=\"#\" \/><\/div>\n<\/div>\n<\/div>\n<\/div>\n<\/div>\n<\/div>\n<div class=\"col-sm-6\">\n<div style=\"overflow: hidden;\">\n<div id=\"carousel-example-generic2\" class=\"carousel slide\" data-ride=\"carousel\">\n<div class=\"carousel-inner\">\n<div class=\"active item\">\n<div class=\"modal-carousel-hybrid-source-image-wrapper screenshot2\"><img class=\"modal-carousel-hybrid-source-image\" src=\"\/wp-content\/uploads\/2017\/01\/audio_digram.png\" alt=\"...\" data-target=\"#myModal\" data-toggle=\"modal\" \/><\/div>\n<\/div>\n<\/div>\n<\/div>\n<div id=\"myModal\" class=\"fade modal\" tabindex=\"-1\">\n<div class=\"modal-dialog\">\n<div class=\"modal-content\">\n<div id=\"modal-carousel-hybrid-popup\" class=\"modal-body\"><img src=\"#\" \/><\/div>\n<\/div>\n<\/div>\n<\/div>\n<\/div>\n<\/div>\n<div class=\"col-sm-6\">\n<div style=\"overflow: hidden;\" align=\"center\">\n<div id=\"carousel-example-generic2\" class=\"carousel slide\" data-ride=\"carousel\">\n<div class=\"carousel-inner\">\n<div class=\"active item\">\n<div class=\"modal-carousel-hybrid-source-image-wrapper screenshot3\"><img class=\"modal-carousel-hybrid-source-image\" src=\"\/wp-content\/uploads\/2017\/01\/txt_digram.png\" alt=\"...\" data-target=\"#myModal\" data-toggle=\"modal\" \/><\/div>\n<\/div>\n<\/div>\n<\/div>\n<div id=\"myModal\" class=\"fade modal\" tabindex=\"-1\">\n<div class=\"modal-dialog\">\n<div class=\"modal-content\">\n<div id=\"modal-carousel-hybrid-popup\" class=\"modal-body\"><img src=\"#\" \/><\/div>\n<\/div>\n<\/div>\n<\/div>\n<\/div>\n<\/div>\n<\/div>\n<h3>Colors<\/h3>\n<p>Based on the description above everything should be grayscale, however, there are different colors visible on the screenshots above. Veles uses them to add just a little bit of information to the visualization. If a digram is found predominantly at the beginning of the file the corresponding point will be more yellow. If it&#8217;s more common at the end it will be blue. If it&#8217;s evenly distributed or most common in the middle of the file the point will be white.<\/p>\n<h2 style=\"margin-top: 45px;\">Layered digram<\/h2>\n<p>In this visualization we divide the file into 256 evenly sized parts. For each of this parts we calculate a digram visualization. We display all of those visualizations on top of each other to create a 3d cube showing how digram distribution changes through the file.<\/p>\n<div class=\"row\">\n<div class=\"col-sm-6\">\n<div style=\"overflow: hidden;\">\n<div id=\"carousel-example-generic2\" class=\"carousel slide\" data-ride=\"carousel\">\n<div class=\"carousel-inner\">\n<div class=\"active item\">\n<div class=\"modal-carousel-hybrid-source-image-wrapper screenshot1\">\n<img loading=\"lazy\" class=\"modal-carousel-hybrid-source-image\" src=\"\/wp-content\/uploads\/2017\/01\/elf_layered.png\" alt=\"...\" width=\"1443\" height=\"983\" data-target=\"#myModal\" data-toggle=\"modal\" \/>\n<\/div>\n<\/div>\n<\/div>\n<\/div>\n<div id=\"myModal\" class=\"fade modal\" tabindex=\"-1\">\n<div class=\"modal-dialog\">\n<div class=\"modal-content\">\n<div id=\"modal-carousel-hybrid-popup\" class=\"modal-body\"><img src=\"#\" \/><\/div>\n<\/div>\n<\/div>\n<\/div>\n<\/div>\n<\/div>\n<div class=\"col-sm-6\">\n<div style=\"overflow: hidden;\">\n<div id=\"carousel-example-generic2\" class=\"carousel slide\" data-ride=\"carousel\">\n<div class=\"carousel-inner\">\n<div class=\"active item\">\n<div class=\"modal-carousel-hybrid-source-image-wrapper screenshot2\"><img class=\"modal-carousel-hybrid-source-image\" src=\"\/wp-content\/uploads\/2017\/01\/audio_layered.png\" alt=\"...\" data-target=\"#myModal\" data-toggle=\"modal\" \/><\/div>\n<\/div>\n<\/div>\n<\/div>\n<div id=\"myModal\" class=\"fade modal\" tabindex=\"-1\">\n<div class=\"modal-dialog\">\n<div class=\"modal-content\">\n<div id=\"modal-carousel-hybrid-popup\" class=\"modal-body\"><img src=\"#\" \/><\/div>\n<\/div>\n<\/div>\n<\/div>\n<\/div>\n<\/div>\n<div class=\"col-sm-6\">\n<div style=\"overflow: hidden;\" align=\"center\">\n<div id=\"carousel-example-generic2\" class=\"carousel slide\" data-ride=\"carousel\">\n<div class=\"carousel-inner\">\n<div class=\"active item\">\n<div class=\"modal-carousel-hybrid-source-image-wrapper screenshot3\"><img class=\"modal-carousel-hybrid-source-image\" src=\"\/wp-content\/uploads\/2017\/01\/txt_layered.png\" alt=\"...\" data-target=\"#myModal\" data-toggle=\"modal\" \/><\/div>\n<\/div>\n<\/div>\n<\/div>\n<div id=\"myModal\" class=\"fade modal\" tabindex=\"-1\">\n<div class=\"modal-dialog\">\n<div class=\"modal-content\">\n<div id=\"modal-carousel-hybrid-popup\" class=\"modal-body\"><img src=\"#\" \/><\/div>\n<\/div>\n<\/div>\n<\/div>\n<\/div>\n<\/div>\n<\/div>\n<h2 style=\"margin-top: 45px;\">Trigram<\/h2>\n<p>This view is most easily explained by relating it to the digram view. In fact it&#8217;s exactly the same thing, except this time we use 3\u2011byte sequences (trigrams) to create a 3d cube. Each byte in the file is used 3 times: once as a coordinate on x-axis, once on y-axis and once on z-axis. The meaning of different colors is also the same as in digrams &#8211; yellow for trigrams most common in the beginning of the file, blue for those found mostly at the end.<\/p>\n<h3>Shapes<\/h3>\n<p>Trigram and layered digram views give user the option to change the shape between cube, cylinder and sphere. All those views present exactly the same data. Each point is represented by 3 values (trigram or digram + layer number). By switching between <a href=\"https:\/\/en.wikipedia.org\/wiki\/Cartesian_coordinate_system\" target=\"_blank\" rel=\"noopener\">cartesian<\/a>, <a href=\"https:\/\/en.wikipedia.org\/wiki\/Cylindrical_coordinate_system\" target=\"_blank\" rel=\"noopener\">cylindrical<\/a> and <a href=\"https:\/\/en.wikipedia.org\/wiki\/Spherical_coordinate_system\" target=\"_blank\" rel=\"noopener\">spherical<\/a> coordinate systems we get different graphical representation of the data.<\/p>\n<style>\n.col-sm-6 {\n  padding-bottom: 30px;\n}\n.screenshot4:after {\n  content: \"Cartesian coordinate system\";\n}\n.screenshot5:after {\n  content: \"Cylindrical coordinate system\";\n}\n.screenshot6:after {\n  content: \"Spherical coordinate system\";\n}\n<\/style>\n<div class=\"row\">\n<div class=\"col-sm-6\">\n<div style=\"overflow: hidden;\">\n<div id=\"carousel-example-generic2\" class=\"carousel slide\" data-ride=\"carousel\">\n<div class=\"carousel-inner\">\n<div class=\"active item\">\n<div class=\"modal-carousel-hybrid-source-image-wrapper screenshot4\">\n<img loading=\"lazy\" class=\"modal-carousel-hybrid-source-image\" src=\"\/wp-content\/uploads\/2017\/01\/elf_cube.png\" alt=\"...\" width=\"1443\" height=\"983\" data-target=\"#myModal\" data-toggle=\"modal\" \/>\n<\/div>\n<\/div>\n<\/div>\n<\/div>\n<div id=\"myModal\" class=\"fade modal\" tabindex=\"-1\">\n<div class=\"modal-dialog\">\n<div class=\"modal-content\">\n<div id=\"modal-carousel-hybrid-popup\" class=\"modal-body\"><img src=\"#\" \/><\/div>\n<\/div>\n<\/div>\n<\/div>\n<\/div>\n<\/div>\n<div class=\"col-sm-6\">\n<div style=\"overflow: hidden;\">\n<div id=\"carousel-example-generic2\" class=\"carousel slide\" data-ride=\"carousel\">\n<div class=\"carousel-inner\">\n<div class=\"active item\">\n<div class=\"modal-carousel-hybrid-source-image-wrapper screenshot5\"><img class=\"modal-carousel-hybrid-source-image\" src=\"\/wp-content\/uploads\/2017\/01\/elf_cylinder.png\" alt=\"...\" data-target=\"#myModal\" data-toggle=\"modal\" \/><\/div>\n<\/div>\n<\/div>\n<\/div>\n<div id=\"myModal\" class=\"fade modal\" tabindex=\"-1\">\n<div class=\"modal-dialog\">\n<div class=\"modal-content\">\n<div id=\"modal-carousel-hybrid-popup\" class=\"modal-body\"><img src=\"#\" \/><\/div>\n<\/div>\n<\/div>\n<\/div>\n<\/div>\n<\/div>\n<div class=\"col-sm-6\">\n<div style=\"overflow: hidden;\" align=\"center\">\n<div id=\"carousel-example-generic2\" class=\"carousel slide\" data-ride=\"carousel\">\n<div class=\"carousel-inner\">\n<div class=\"active item\">\n<div class=\"modal-carousel-hybrid-source-image-wrapper screenshot6\"><img class=\"modal-carousel-hybrid-source-image\" src=\"\/wp-content\/uploads\/2017\/01\/elf_sphere.png\" alt=\"...\" data-target=\"#myModal\" data-toggle=\"modal\" \/><\/div>\n<\/div>\n<\/div>\n<\/div>\n<div id=\"myModal\" class=\"fade modal\" tabindex=\"-1\">\n<div class=\"modal-dialog\">\n<div class=\"modal-content\">\n<div id=\"modal-carousel-hybrid-popup\" class=\"modal-body\"><img src=\"#\" \/><\/div>\n<\/div>\n<\/div>\n<\/div>\n<\/div>\n<\/div>\n<\/div>\n<hr \/>\n<h2>Sampling<\/h2>\n<p>Performance\u2011wise it&#8217;s not always feasible to visualize the whole file. For larger files the amount of data to process is just too much. However, the binary visualization is most useful when working with large files.<\/p>\n<p>In Veles we handle this problem by visualizing a sample of data instead of the whole file. Sampling is done before any further processing. From the point of view of visualization the sampled data is the byte stream we&#8217;re analysing. Increasing sample size improves accuracy of visualization and limits the chance of introducing artifacts at the cost of performance. Depending on available hardware we normally use sample size between 1MB\u00a0and\u00a010MB.<\/p>\n<p>Note that Veles will not sample files smaller than the sample size set by user. In that case it just shows the whole file. Additionally we provide an option to disable sampling in UI. Selecting this option will lead to poor frame rates or even crash Veles when working on very large files!<\/p>\n<h3>Uniform random sampling<\/h3>\n<p>Let N = sample size. The default sampling algorithm randomly picks sqrt(N) continuous byte sequences of size sqrt(N) from file. This approach is a compromise between the need for taking data from multiple different places in the file and the requirement for the sample to represent actual byte sequences for n\u2011gram visualizations.<\/p>\n<hr \/>\n<h2>Minimap<\/h2>\n<p>The panel on the left side of the visualization is minimap. It shows a simple visualization of the whole file. You can slide the 2 bars to select a (continuous) part of the file. Only the selected part of the file is visualized in the main part of the window.<\/p>\n<p>You can add additional minimaps if you want to zoom in a specific part of the file. Leftmost minimap always shows the whole file. Each consecutive minimap shows only the selected range of the previous one. Using 3 or 4 minimaps you can easily select just a specific few hundred of bytes out of a very large file.<\/p>\n<h3>Minimap modes<\/h3>\n<p>Minimap divides the file into parts of equal size and represents each part with a single <a href=\"https:\/\/en.wikipedia.org\/wiki\/Texel_(graphics)\" target=\"_blank\" rel=\"noopener\">texel<\/a>. The file is represented left\u2011to\u2011right and top\u2011to\u2011bottom. In default (green) mode the value of each texel is calculated based on average value of bytes in the corresponding part of the file. Alternative mode (red) shows the <a href=\"https:\/\/en.wiktionary.org\/wiki\/Shannon_entropy\" target=\"_blank\" rel=\"noopener\">Shannon entropy<\/a> instead.<\/p>\n<hr \/>\n<p>Check out our earlier post to see how we can identify different parts of a real file (libc.so from Ubuntu 14.04) and find out CPU architecture of the machine: <a href=\"https:\/\/codisec.com\/binary-data-visualization\/\" target=\"_blank\" rel=\"noopener\">Binary data visualization<\/a>.<\/p>\n<hr \/>\n<p>Veles visualizations were inspired by Christopher Domas&#8217; talk from Derbycon 2012. Check out the video of his presentation <a href=\"https:\/\/www.youtube.com\/watch?v=4bM3Gut1hIk\" target=\"_blank\" rel=\"noopener\">https:\/\/www.youtube.com\/watch?v=4bM3Gut1hIk<\/a>.<\/p>\n","protected":false},"excerpt":{"rendered":"<p>Veles visualizations are purely statistical representations of binary data. We take a sequence of bytes and visualize correlations between certain values. It doesn&#8217;t matter if it&#8217;s an executable file, a picture or a disk image &#8211; from the perspective of<span class=\"ellipsis\">&hellip;<\/span> <a href=\"https:\/\/codisec.com\/binary-visualization-explained\/\"><\/p>\n<div class=\"read-more\">Read more &#8250;<\/div>\n<p><!-- end of .read-more --><\/a><\/p>\n","protected":false},"author":5,"featured_media":0,"comment_status":"closed","ping_status":"closed","sticky":false,"template":"","format":"standard","meta":[],"categories":[22,24],"tags":[],"yoast_head":"<!-- This site is optimized with the Yoast SEO plugin v16.1.1 - https:\/\/yoast.com\/wordpress\/plugins\/seo\/ -->\n<link rel=\"canonical\" href=\"https:\/\/codisec.com\/binary-visualization-explained\/\" \/>\n<meta property=\"og:locale\" content=\"en_US\" \/>\n<meta property=\"og:type\" content=\"article\" \/>\n<meta property=\"og:title\" content=\"Binary visualization explained - CodiSec\" \/>\n<meta property=\"og:description\" content=\"Veles visualizations are purely statistical representations of binary data. We take a sequence of bytes and visualize correlations between certain values. It doesn&#8217;t matter if it&#8217;s an executable file, a picture or a disk image &#8211; from the perspective of&hellip; Read more &#8250;\" \/>\n<meta property=\"og:url\" content=\"https:\/\/codisec.com\/binary-visualization-explained\/\" \/>\n<meta property=\"og:site_name\" content=\"CodiSec\" \/>\n<meta property=\"article:published_time\" content=\"2017-01-13T17:48:31+00:00\" \/>\n<meta property=\"article:modified_time\" content=\"2023-03-22T15:29:58+00:00\" \/>\n<meta property=\"og:image\" content=\"https:\/\/codisec.com\/wp-content\/uploads\/2017\/01\/digram_example.png\" \/>\n<meta name=\"twitter:card\" content=\"summary\" \/>\n<meta name=\"twitter:label1\" content=\"Est. reading time\">\n\t<meta name=\"twitter:data1\" content=\"5 minutes\">\n<script type=\"application\/ld+json\" class=\"yoast-schema-graph\">{\"@context\":\"https:\/\/schema.org\",\"@graph\":[{\"@type\":\"WebSite\",\"@id\":\"https:\/\/codisec.com\/#website\",\"url\":\"https:\/\/codisec.com\/\",\"name\":\"CodiSec\",\"description\":\"PENETRATION TESTING  AND MALWARE ANALYSIS\",\"potentialAction\":[{\"@type\":\"SearchAction\",\"target\":\"https:\/\/codisec.com\/?s={search_term_string}\",\"query-input\":\"required name=search_term_string\"}],\"inLanguage\":\"en-US\"},{\"@type\":\"ImageObject\",\"@id\":\"https:\/\/codisec.com\/binary-visualization-explained\/#primaryimage\",\"inLanguage\":\"en-US\",\"url\":\"https:\/\/codisec.com\/wp-content\/uploads\/2017\/01\/digram_example.png\",\"contentUrl\":\"https:\/\/codisec.com\/wp-content\/uploads\/2017\/01\/digram_example.png\",\"width\":1600,\"height\":1404},{\"@type\":\"WebPage\",\"@id\":\"https:\/\/codisec.com\/binary-visualization-explained\/#webpage\",\"url\":\"https:\/\/codisec.com\/binary-visualization-explained\/\",\"name\":\"Binary visualization explained - CodiSec\",\"isPartOf\":{\"@id\":\"https:\/\/codisec.com\/#website\"},\"primaryImageOfPage\":{\"@id\":\"https:\/\/codisec.com\/binary-visualization-explained\/#primaryimage\"},\"datePublished\":\"2017-01-13T17:48:31+00:00\",\"dateModified\":\"2023-03-22T15:29:58+00:00\",\"author\":{\"@id\":\"https:\/\/codisec.com\/#\/schema\/person\/76ede82820a90a251ac07a0dc4f9fd87\"},\"breadcrumb\":{\"@id\":\"https:\/\/codisec.com\/binary-visualization-explained\/#breadcrumb\"},\"inLanguage\":\"en-US\",\"potentialAction\":[{\"@type\":\"ReadAction\",\"target\":[\"https:\/\/codisec.com\/binary-visualization-explained\/\"]}]},{\"@type\":\"BreadcrumbList\",\"@id\":\"https:\/\/codisec.com\/binary-visualization-explained\/#breadcrumb\",\"itemListElement\":[{\"@type\":\"ListItem\",\"position\":1,\"item\":{\"@type\":\"WebPage\",\"@id\":\"https:\/\/codisec.com\/\",\"url\":\"https:\/\/codisec.com\/\",\"name\":\"Home\"}},{\"@type\":\"ListItem\",\"position\":2,\"item\":{\"@type\":\"WebPage\",\"@id\":\"https:\/\/codisec.com\/binary-visualization-explained\/\",\"url\":\"https:\/\/codisec.com\/binary-visualization-explained\/\",\"name\":\"Binary visualization explained\"}}]},{\"@type\":\"Person\",\"@id\":\"https:\/\/codisec.com\/#\/schema\/person\/76ede82820a90a251ac07a0dc4f9fd87\",\"name\":\"Maciej Pytel\",\"image\":{\"@type\":\"ImageObject\",\"@id\":\"https:\/\/codisec.com\/#personlogo\",\"inLanguage\":\"en-US\",\"url\":\"https:\/\/secure.gravatar.com\/avatar\/fe6f19b992c8c1d2877f0eee7a7302ec?s=96&d=mm&r=g\",\"contentUrl\":\"https:\/\/secure.gravatar.com\/avatar\/fe6f19b992c8c1d2877f0eee7a7302ec?s=96&d=mm&r=g\",\"caption\":\"Maciej Pytel\"}}]}<\/script>\n<!-- \/ Yoast SEO plugin. -->","_links":{"self":[{"href":"https:\/\/codisec.com\/wp-json\/wp\/v2\/posts\/13760"}],"collection":[{"href":"https:\/\/codisec.com\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/codisec.com\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/codisec.com\/wp-json\/wp\/v2\/users\/5"}],"replies":[{"embeddable":true,"href":"https:\/\/codisec.com\/wp-json\/wp\/v2\/comments?post=13760"}],"version-history":[{"count":63,"href":"https:\/\/codisec.com\/wp-json\/wp\/v2\/posts\/13760\/revisions"}],"predecessor-version":[{"id":14996,"href":"https:\/\/codisec.com\/wp-json\/wp\/v2\/posts\/13760\/revisions\/14996"}],"wp:attachment":[{"href":"https:\/\/codisec.com\/wp-json\/wp\/v2\/media?parent=13760"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/codisec.com\/wp-json\/wp\/v2\/categories?post=13760"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/codisec.com\/wp-json\/wp\/v2\/tags?post=13760"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}