Learning Python - Crypto Challenge 5

This challenge involves writing a repeating key XOR encryption method, and encrypt the following string: 

Burning 'em, if you ain't quick and nimble
I go crazy when I hear a cymbal

Should produce:


Repeating key XOR

I actually over-engineered my solution to the fixed length XOR Challenge 2, and have already solved this problem. In the 'encipher_xor' method I use the modulus operator with the key length to repeat over the key, if the key length is less than the plain text length.


For the sake of completeness here is my solution:

#!/usr/bin/env python
'''Implement repeating-key XOR'''

def encipher_xor(plain, key):
        cipher = bytearray()
        for i in xrange(len(plain)):
            cipher.append(chr(plain[i] ^ key[i%len(key)]))
        return cipher

def main():    
    p = bytearray("Burning 'em, if you ain't quick and nimble\nI go crazy when I hear a cymbal")
    k = bytearray('ICE')
    c = encipher_xor(p, k)
    encoded = str(c).encode('hex')
    print 'plain: {0}'.format(p)
    print 'cipher: {0}'.format(encoded)
    assert(encoded == '0b3637272a2b2e63622c2e69692a23693a2a3c6324202d623d63343c2a26226324272765272a282b2f20430a652e2c652a3124333a653e2b2027630c692b20283165286326302e27282f')

if __name__ == "__main__":

by IAN

Learning Python - Crypto Challenge 4

Challenge 4 builds on top of the previous challenge, and involves finding the single string contained within this file that has been encrypted with single byte XOR.

Detecting Single Byte XOR

The steps for solving this challenge are :

  • Open and read the cipher file
  • for each line string in the file use the 'solve_single_key_xor' method from the previous challenge, to find a scoring value and the best possible key
  • Select the highest scoring string from the previous stage result

#!/usr/bin/env python
'''Single-byte XOR cipher'''

import re
from s1c3 import solve_single_key_xor

def detect_single_xor(lines):
    res = []
    for line in lines:
        line = bytearray(re.sub('[\n]','', line).decode('hex'))
    return (max(res, key=lambda x: x[0]))  

def main():        
    with open('s1c4.txt', 'r') as f:
        lines = f.readlines()

    print detect_single_xor(lines)

if __name__ == "__main__":

So a relatively simple extension to challenge 3 but it does help to consolidate reading hex encoded strings from a file, and converting to byte arrays. The lambda functions in Python are pretty powerful and aid in writing neat, if not so readable code.

by IAN


NotificationWindow 'ghost' blocking other GUI controls

Tag Cloud Block
This is an example. Double-click here and select a page to create a cloud of its tags or categories. Learn more

We are utilising the NotificationWindow class in our Silverlight application to do a customised version of the popup message ('toast') to the user, in the system area of the desktop.

Through testing it was discovered that the NotificationWindow will leave a 'ghost' in the visual tree that will block the top-left of the application (that runs out of browser). 

That is, if you have a 100 by 200 NotificationWindow appearing in the bottom right of the screen, then the top left of the application enjoys an equivalent 100 by 200 dead-zone that will not activate the mouse-over or click events as if a piece of glass was in the way.

Once the message is dismissed or times out, normal service resumes

There was another person who had the issue on the web, though when the bug was formally logged it was closed as a "Won't Fix".

However, there is a workaround.  I found that you can carefully apply


You could apply it to the Grid or StackPanel that holds all of the content of the NotificationWindow's template but that would mean that any controls on the form are now rendered non-interactive.  Great for just text.  Not so when you need a button.

In our case we had a small X to close the notification manually, so I applied IsHitTestVisible throughout the template but just ignored enough for the X button to remain clickable.

I would imagine that there is now a smaller dead-zone, the same size as that clickable button, somewhere equivalent up in the top left corner - but between me and our tester we have yet to find it.  The reason we haven't found it would be because it doesn't block anything with the mouse-over or click event.

There you go - a Silverlight post for August 2015



The Dilemma Of Updating

In the day job I want to be using the latest tools and versions of software.  The business has no need to be cutting edge (technology choices were made and final about 3 years ago) and the working environment makes it pretty clear that you use what you got and there won't be updates to the environment for the foreseeable future.

As a team, we bemoan not having the latest tools and we believe that having the latest and greatest gives us everything we will need - plus it keeps us more cutting edge and we all want that in our technical jobs, right?  No one wants to be too far behind the curve.

In my side project where I have total control, I too have made a cut off and I now fear the update and what it may bring.

I understand that pain and dilemma of updating now.

Visual Studio 2013 has worked great so far - I don't need 2015.... I think!

The ASP.NET MVC I started with is at version 5 - I do not believe that I will need version 6 as (to me, in a nutshell) it's all Azure focused, the "Cloud Optimized Framework" and all that.  Staying with 5 will be fine...

I appreciate that I am using a particular version of the Telerik controls.  For what I have experienced of them, they work fine.  I haven't encountered any shortfall in what they provide, nor do I need any of the newer controls they have created.  Plus, from previous experience, I am fully aware that a Telerik upgrade can have a breaking change - but this was a breaking change because we didn't update for 2 whole years:  The longer you leave it the more painful it could be.

No doubt SQL Server (Express) 2016 will be available one day too.  I really do not want to touch that! I mean, the database is the essential part of the application.

Having time to really evaluate the tools is one thing; double checking that they haven't broken something that worked previously is something else.

Upgrading should be done in a controlled manner to mitigate any risk of unknown consequences of change.  What are the consequences and mitigations of upgrading (or not)?  How do you assess the risk?  For the bugs fixed in the latest release, new unknown bugs will come along.  Is it a security risk to upgrade - likewise is it a security risk to NOT upgrade?

So to reiterate, I understand that pain and dilemma of updating now.  I know that I would feel guilty if I had to explain to my customer that things had gone wrong because of an upgrade.  I know I would kick myself because of the old adage "If it ain't broke, don't try and fix it".  We software-people get it but my non-technical customer would not be as 'sympathetic' to the desire of the having and using latest tools because they are meant to be the best.

When you are solely responsible for keeping your customer happy, what do you think you would do?


A Small Victory - Entity Framework not regenerating on 'Update Model From Database...'

For a while, I put up with something that I should have nailed a long time ago.

When I added a new view or table to SQL Server and then pulled it into Entity Framework (using the 'Update Model from Database...' mechanism on the EDMX canvas in Visual Studio)  a previous view that was working would 'break'.

I discovered the problem when I rebuilt the solution: My ViewModel classes would break as the previously working Entities had not pulled in again.  That is because the auto-generated Entity class that I had previously mapped to my view model class had vanished.

Looking in the EDMX file, I would find this error:

warning 6002: The table/view '{mydb}.dbo.{my_view}' does not have a primary key defined. The key has been inferred and the definition was created as a read-only table/view

So I put up with it for a while as source control meant I could refer to the last time that block of code worked.. So I would Cut (from working code).. Paste (over the broken code).. Recompile; "It's fixed again"; move on.

Then the time came where I dedicated an afternoon to fixing the weak view problem as it was a poor way to continue working.

The answer was making sure that I had a 'proper table' with a unique index/id included in the SQL.

I had a view, that was based upon non-unique indexes - I had customer equipment in a JOIN with service histories based upon the customer ID. 


When I included the Customer Table that OWNED the unique customer ID then my problem went away


Without it, it is as if Entity Framework had no proper hook into the data.

Well that is how I am going to understand it in my head.


A Small Victory - The "IIS APPPOOL\DefaultAppPool" user

I had a Small Victory after about a week (on and off) of trying to work out what was preventing access to my ASP.Net application's data (in SQL Express).  I was attempting a 'practise run' of an installation using a virtual machine.

I had deployed the application to IIS Express fine, using the 'Publish' mechanism in Visual Studio.  I had followed everything I needed to do with IIS and ASP.NET, such as:

  • Activating the Application Development Features for ASP.NET 4.5
  • Using aspnet_regiis.exe  and setting DefaultAppPool
  • Configuring the Windows Firewall to allow www access
  • Configuring web.config to point to the correct database location

All went very smoothly - or so I thought.  At runtime, I had my application appear but when ever I queried the database no data from the database would populate the screen.

I was baffled.  I wasn't receiving any obvious error.

This is when my Google-Fu appeared to let me down - I had no idea what to search for and what I did find talked a lot about 'LocalDB' deployment that I wasn't using.  So I probably tended to skim those pages..

Other pages suggested using SQL Authentication instead, but I did not want to do that.

I tried reinstalling everything, destroying and recreating the VM a number of times (as I had hacked the configurations over and over)

Eventually I found help on a post on stackoverflow, when somebody suggested looking at the Event Viewer and looking for errors.

Sure enough, I FINALLY found my problem - I had failed to add the IIS APPPOOL\DefaultAppPool user and grant permissions to the users in SQL Express (as the Event Viewer reported that it was being denied access).

In hindsight, I see that Microsoft does allude to a grant script but at the time (for whatever reason) it 'didn't make sense' and I was unsure how to relate what I was doing to the advice given (ie. I was not going to grant db_owner to a user role).

In the calm light of day, reading the advice makes perfect sense now I know what it means and how it affects things.


Learning Python - Crypto Challenge 3

The first two challenges have shown converting from hexadecimal to base64 encoding and how to XOR strings together so we are now ready to solve our first cipher. Admittedly it is a simple cipher but we will quickly build upon this in the later challenges moving from solving single XOR to arbitary length repeating key XOR.

Single byte XOR cipher

The following text has been encrypted using a single XOR cipher:


We need to solve this cipher by finding which single byte value XOR's with the above cipher to give plain text. We can break this down into two steps, a method that will score an array of bytes on how many of the characters are within the ASCII A-Z and a-z range. Additionally we require a method that will iterate over all possible single byte values [0 to 255], deciphering the cipher-text and passing the possible candidate solution to our scoring method. Once we have a scored all possible solutions we will take the highest score as our answer.

'''Single-byte XOR cipher'''

def score_alphatext(txt):
        scr = filter(lambda x: 'a'<=x<='z' or 'A'<=x<='Z' or x==' ', txt)
        return float(len(scr)) / len(txt)

def solve_single_key_xor(ba):
    res = []
    for i in range(256):
        characters = [chr(c ^ i) for c in ba]
        res.append([score_alphatext(characters), ''.join(characters), i])
    return max(res, key=lambda x: x[0])

def main():
    cipher = '1b37373331363f78151b7f2b783431333d78397828372d363c78373e783a393b3736'
    print solve_single_key_xor(bytearray(cipher.decode('hex')))

if __name__ == "__main__":

by IAN

A Small Victory - The 'google search'

A Small Victory is a series where I have an opportunity for me to explain how I did something.

It is most probably something that is trivial; something a lot of people would take for granted and find easy and has been done a thousand times before elsewhere - but for me it is a small victory.

Today's victory is how I gave my customer a 'Google search' for their data grid:  "I want to be able to type in a bit of data - a bit of their name or address or telephone number - and it will give me that list I can click and load their data".

Using Telerik's ASP.NET MVC AutoComplete control I added the control to the HTML

<div class="quick-find-section">
    <label for="contacts">Find contact:</label>
      .HtmlAttributes(new { style = "width:1000px" })
      .DataSource(source => source
        .Read(read => read.Action("QuickSearchContacts", "Contacts"))
      .Events(e =>

The important bits here are

  • The QuickSearchContacts()method call made to the Contacts controller : this provides the .DataSource for the list - that is, when the user starts to type in the search information then this is the list of information they will start to narrow down the more they type.
  • The onQuickSelect() function call in the View's Javascript : this tells the grid to filter its dataSource to equivalent of what the user selected (filtered by id).  The Javascript is just below.

function onQuickSelect(e) {
  var dataItem = this.dataItem(e.item.index());
  var contactId = dataItem.Contact_id;
  var grid = $("#Grid").data("kendoGrid");

  grid.dataSource.filter({field:"Contact_id",operator: "eq",value:contactId});

Therefore the  user types in their fragment of information, the list drops down and, as they type, the list shortens to a point where the user will select an item.  This select event filters their grid of many customers to a grid of just one item - the one item that the user selected in the Autocomplete.

Below is the code from Controller.QuickSearchContacts().  It takes the string each time the user types a character into the search.  As long as you have 2 or more characters it will provide a list of Contacts information.  Each ContactsViewModel has a convenience string property (QuickSearch) that has their searchable details e.g. "Bob Brown 12 Meadow View Hurley 07987 234434 bob@emailcity.org" 

public JsonResult QuickSearchContacts(string text)
    IEnumerable contacts = GetVMContacts();

    if (!string.IsNullOrEmpty(text) && text.Length > 2)
        contacts = contacts.Where(p => p.QuickSearch.Contains(text));

    return Json(contacts, JsonRequestBehavior.AllowGet);



The 'Chicken and Egg' of Cleaning Data

Currently, outside of the day job, I am trying to help someone modernise their business application.

As a high concept - it's a customer database with job/visit scheduling.  So a customer will call the firm requesting some work and the firm will then attend to that job when they schedule it in.

The currently adopted technology is an Access 2003 Database Application from quite a few years ago and the database design (ahem) is something 'I am not used to'.

I am putting together an SQL Server (Express) for them and exporting a table or two at a time from the Access database as Excel files and importing that Excel into SQL Server:  I have to do it this way because if I try to export directly the data fails it's own constraints(!)  Then with Entity Framework and ASP.NET MVC I am producing a new application.  This is also an opportunity for me to learn ASP.NET MVC etc.

Being an outsider from the business and given the job where they consider me the expert who knows what to do, I have full reign on the layout and presentation of the data - and this is good, because when I present my latest demo and deliver the next 'cut' it highlights the inaccuracies and mistakes in their legacy data.  It also highlights the backlog of data they don't even want any more.

The whole purpose of this technology refresh was to allow them to work smarter and easily analyse their own data and to plan their work efficiently.

But as the application is gradually coming together, I have a situation where the client is keen to start cleaning their data.  But to allow them easier cleaning of the data, I need to get the application more complete. Hence the Chicken and Egg title to this post.

The problem is there never seems to be a good time to clean the data:  The hypothetical ideal time would be the day when the old system shuts off, the data is taken off - it is cleaned - and then imported to the new system just before the new system goes live.  But cleaning data takes a long time and we need that good data NOW!

The best thing I think I can suggest is that they keep using the new system to find the bad data the their current system.  They can then continue to use the current system for cleaning.  Then, when I take their latest MDB, I can move the cleaner data into my developing system.  I know this is quite a manual process, but I have only recently learned that SSIS may have helped all along (but I haven't got the time to learn that too).


Learning Python - Crypto Challenge 2

Fixed XOR

Challenge two consists of XOR enciphering one hexidecimal encoded string with another hexidecimal encoded string.


XOR'ed  686974207468652062756c6c277320657965
gives   746865206b696420646f6e277420706c6179

Firstly we need to convert from the hexidecimal encoding to simple byte arrays, then we can iterate over the string XOR'ing the byte array values together to give our result.

#!/usr/bin/env python
'''Fixed XOR'''

def encipher_xor(plain, k):
        cipher = bytearray()
        for i in xrange(len(plain)):
        return cipher

def main():
    # decode hex encoding to array of bytes 
    barray1 = bytearray('1c0111001f010100061a024b53535009181c'.decode('hex'))
    barray2 = bytearray('686974207468652062756c6c277320657965'.decode('hex'))

    xor = encipher_xor(barray1,barray2)
    print 'xor: {0}'.format(xor)

    # check the answer
    assert(xor == '746865206b696420646f6e277420706c6179'.decode('hex'))

if __name__ == "__main__":

So analysing the code ... we have the usual python 'shebang' and module name checking with a main entry point. Two byte arrays are setup to hold the decoded hexadecimal strings. Finally we have the XOR method, this is a little more complicated than it needs to be as it will account for enciphering different length byte arrays {needed in further challenges) and returning the resultant bytearray. Note the need to decode the expected result with the actual result as the 'encipher_xor'  method returns a bytearray.

Now with the basics of string encoding and XOR operation under our belts we can move onto some actual deciphering challenges...

by IAN


Learning Python - Crypto Challenge 1

Lets start with the first challenge, nothing too stretching but it provides a good opportunity to deal with String encoding.

Challenge One - Convert hex to base64

The string:


Should produce:


So lets make that happen. We are trying to convert a hexidecimal encoded string into base64 represetation, so to start we need to convert the hexidecimal encoding into a collection of bytes. Next we have a few options we can use the String encode methods or the base64 module to do the base 64 encoding. In this case I will use the base64 module. 

#!/usr/bin/env python
'''Convert hex to base64'''

import base64

def main():    
    b16 = bytearray('49276d206b696c6c696e6720796f757220627261696e206c696b65206120706f69736f6e6f7573206d757368726f6f6d'.decode('hex'))    
    b64 = base64.b64encode(b16)

    print b64
    assert(b64 == 'SSdtIGtpbGxpbmcgeW91ciBicmFpbiBsaWtlIGEgcG9pc29ub3VzIG11c2hyb29t')

if __name__ == "__main__":

by IAN

Learning Python - The Matasano Crypto Challenges

I find it difficult picking up a new language, the dithering over syntax and which library or module should I use generally find me gravitating back to my languages of choice, C# or C++. However I need to learn a solid scripting language and there is nothing like a deadline to focus the mind. I start my Offensive Security 'Penetration testing with Kali linux' training in a few weeks and it is about time I brushed up on my Python skills. I have used Python a few years ago so scripting languages are not a mystery but I'm rusty and I find I pick up a language so much easier by doing rather than just reading. 

Hence I shall be slowly working my way through the excellent Matasano Cryptography Challenges.

by IAN

Asus Transformer TF101 Development Setup

I have an original Asus Transformer TF101 (with the keyboard), that I have always tinkered around with from time to time.

Recently, after a long stint of it not actually being used, I looked up the latest and greatest ROM I could find - I loaded it with KatKiss (to get the latest Android features and a speed boost), and then I looked to use it as an alternative development platform.

After attending DevWeek 2014 in London back in April, I was made aware of the AIDE - the Android IDE app - in the session "Options for Getting Into Android" with Mark Murphy.

I have always wanted something accessible and quick to just get coding on.  In the past I have purchased a laptop for such a task, but recently it died :( .  I have my "main" PC for Visual Studio (with an SSD, to keep things bearable!) and now I have my TF101 for other stuff:

In addition to AIDE, last week I also installed an AMP equivalent called "Bit Web Server" and "AWD" (Android Web Developer) - an IDE for PHP/HTML/CSS/JS.  I purchased Bit Web Server because it was only about £1.50.  AWD was free, but I unlocked for £3.50 - the unlock giving you nice things like code formatting.

On my iPad I have a copy of  Learning PHP, MySQL, JavaScript, CSS & HTML5 by Robin Nixon - so I am all good to go.

It means that I can sit in bed with a "cool running" laptop-equivalent to do some development.  No exhaust ports getting hot or fans whirring away to distract the wife.

Here are the Google Play store links for the apps I described earlier: