Goodbye AussieWeb. Hello SSW.

After more then 13 years, I have made the decision to move on from my Software Engineering role with Monte Huebsch and team at AussieWeb and take on a Senior Software Architect position with Adam Cogan and his team at SSW.

Goodbye AussieWeb

AussieWeb has been my home for more then a decade. When I started, AussieWeb was a small business, with only 4 permanent part-time employees; .Net was still in it's early releases (1.0 and 1.1), and the main business was Website Development & Hosting (partnered with Lloyd Ernst's WebCentral).

Today: AussieWeb employees 10 full time staff on generous salary & benefits packages; while the business itself has transformed from a Website Development & Hosting provider to one of the leading Google AdWords Partners in the Asia-Pacific Region. The core technology stack as I am leaving it, is .Net 4.5.1, SQL Server 2008R2 (and SQL 2014SP1), Entity Framework Code First, and ASP.NET MVC 5.

My time at AussieWeb has allowed me to do things that I would never have been able to do if I had taken the normal corporate developer route. I was able to live in different cities & countries, work remotely during a period in time where the idea of "remote" workers was still a new, untested and fraught with dangers and generally follow the path I decided was the most appropriate at the time. Ultimately I had complete flexibility as to when and where I worked; the technology stacks, platforms and devices I worked on/from. It was a dream job, with a steady paycheck. All the benefits of the "Nomadic Entrepreneur"/"Tropical MBA" lifestyle, without the responsibility of having to worry about meeting costs and payroll.

Though ultimately family life has caught up.

Hello SSW

I was introduced to SSW through the Brisbane .Net User Group, to be frank, I was impressed by the level of knowledge displayed by their Brisbane staff. SSW at this point feels like an environment where I can continue to grow my craft and expand my knowledge.

Though my official start date with SSW is July 1st 2015, I have been working with the Brisbane Team in a part time capacity since June 1st 2015. This has allowed me to ease myself into SSW's work-flow and methodologies, though not without some level of pain. It's not always easy to move from having complete freedom to following the rules

To date: I have pushed changes to the SSW Website, Sugar Learning and have started working with Brisbane Catholic Education on one of their internal Information Systems.

So a final Goodbye to my coworkers of 13+ years and AussieWeb, and hello to all of the team at SSW.

How to fix your Python virtualenv after a Homebrew Python upgrade

Do you manage your OS X Python installation with Homebrew? Have you recently upgraded Python?

In my case it was a Python 3 point release; from 3.4.2 to 3.4.3. The downside is that this is enough to invalidate virtualenv's symbolic links.

Example

~: cd ~/src/my_app
~/src/my_app: source venv/bin/activate
[venv] ~/src/my_app: python
dyld: Library not loaded: @executable_path/../.Python
  Referenced from: /Users/jeremycade/src/my_app/env/bin/python
  Reason: image not found
Trace/BPT trap: 5

As I've mentioned the virtualenv symlinks to your Homebrew python installation no longer link to the correct locations. The solution is to delete, and regenerate virtualenv symlinks.

First thing, we need to ensure that the your virtualenv is not active.

[venv] ~/src/my_app: deactivate

Next, delete the offending symlinks.

~/src/my_app: find venv -type l -delete

In this case I've made use of the BSD find command that ships with OS X.

The final step is to recreate your virtualenv.

~src/my_app: virtualenv venv

Connecting to Microsoft SQL Server with python in OS X Yosemite

Thanks in part to my technology ADD, and desire to be a general language polyglot; it is by no means unusual to have a mixed technology stack.

In this case I have a number of python micro-services and command line applications, running on a Mac Mini loaded with OS X Yosemite, that need to communicate with Microsoft's SQL Sever 2008 R2, on top of Windows Server 2008 R2.

While SQL Server is a great Relational Database Management System (RDBMS), it isn't exactly known for it's ease of use when it comes to cross-platform communication. A quick Google Search will return a range of results (and horror stories) on how to get this all to work.

Getting Started

It is assumed that you already have a working Python installation, either in the Python 2.7+ or Python 3.3+ families and Pip.

Additionally, we will need:

Decisions and Rationale

I personally prefer to use Homebrew due to it's simplicity; though I do have a MacBook with Macports installed.

The pymssql vs pyodbc is not one I really care to worry about. Though I generally choose to use pymssql due to better support for MS SQL Stored Procedures, and a less troublesome install. In the past I have had issues installing pyodbc via Pip.

Installing FreeTDS

FreeTDS is a set of *nix libraries, that allow applications to talk to Microsoft SQL Server.

In your typical *nix environment FreeTDS sources configurations from two difference places:

${HOME}/.freetds.conf
/etc/freetds.conf

This is slightly different under OS X, and will vary depending on the package manager used to install FreeTDS. FreeTDS reads the ${HOME}/.freetds.conf before resorting to the global freetds.conf. You can find out what the location of your global freetds.conf by executing the following command:

tsql -C

Your result will be something along these lines:

$ tsql -C
Compile-time settings (established with the "configure" script)
                            Version: freetds v0.91
             freetds.conf directory: /usr/local/Cellar/freetds/0.91_2/etc
     MS db-lib source compatibility: no
        Sybase binary compatibility: no
                      Thread safety: yes
                      iconv library: yes
                        TDS version: 7.1
                              iODBC: no
                           unixodbc: no
              SSPI "trusted" logins: no
                           Kerberos: no

Full details about the FreeTDS configuration files can be found here.

Homebrew

brew install freetds

Homebrew symlinks the global freetds.conf file to:

/usr/local/etc/freetds.conf

Macports

sudo port install freetds

Macports installs the global freetds.conf file to:

/opt/local/etc/freetds/freetds.conf

Testing FreeTDS

tsql -H <host> -p <port> -U <username> -P <password>

If everything is working you will see something along the lines of:

locale is "en_AU.UTF-8"
locale charset is "UTF-8"
using default charset "UTF-8"

My Macports machine presented the following errors:

locale is "en_AU.UTF-8"
locale charset is "UTF-8"
using default charset "UTF-8"
Error 20018 (severity 9):
    Unexpected EOF from the server
    OS error 36, "Operation now in progress"
Error 20002 (severity 9):
    Adaptive Server connection failed
There was a problem connecting to the server.

This was overcome by specifying the TDS Version as an Environment Variable like so:

TDSVER=7.0 tsql -H <host> -p <1433 or custom port> -U <username> -P <password>

Your other option for overcoming this type of error is to specify DNS names within your local freetds.conf file, for example:

[myhost]
    host = dbdev.my-dev.com
    port = 1433
    tds version = 7.0

You can this test this by using the following command:

tsql -S myhost -U <USER> -P <PASSWORD>

Installing pymssql

pymssql is easily installed on your machine via Pip.

pip install pymssql

Once installed you can test the package in a fairly simple manner via Python interactive.

$ python3 -i
Python 3.4.2 (default, Oct 17 2014, 20:25:14) 
[GCC 4.2.1 Compatible Apple LLVM 6.0 (clang-600.0.51)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> import pymssql
>>> conn = pymssql.connect("HOST","USER","PASSWORD","DATABASE")
>>> cursor = conn.cursor()
>>> cursor.execute("select * from dbo.[TABLE]")
>>> for row in cursor:
...     print("row = %r" % (row, ))
... 
row = (1, 'LITERAL', Decimal('123.123'))
row = (2, 'LITERAL', Decimal('123.123'))
row = (3, 'LITERAL', Decimal('123.123'))
row = (4, 'LITERAL', Decimal('123.123'))
row = (5, 'LITERAL', Decimal('123.123'))
row = (6, 'LITERAL', Decimal('123.123'))
>>> conn.close()
>>> exit()

pymssql adheres to the Python DB-API 2.0 specification. Meaning that all of the standard cursor methods are available. It also allows you to use ORM packages like SQLAlchemy. pymssql also allows you to call MS SQL Stored procedures via the cursor.execproc method, which for me is a must have feature, particularly when dealing with odd inflexible (EVERYTHING MUST BE A SP) DBA.

Generating Page Slugs in T-SQL!

I recently came across a situation where I needed to generate page Slugs from some 5,000,000+ records in a MS-SQL database. Unfortunately I didn’t have Remote Desktop or Telnet access to the SQL box which precluded me from running one of my normal C# or Powershell solutions, and the idea of doing all of the processing over the wire just didn’t appeal to me. So I decided the best course of action would be to use T-SQL to generate the Slugs for me.

After posting the question to StackOverflow and doing some searching on Google I came across Pinal Dave’s UDF_ParseAlphaChars function. Luckily it had most of the functionality I required, so it only required minor modifications to get the desired result.

The modified code is below.

 1 CREATE FUNCTION [dbo].[UDF_GenerateSlug]
 2     (
 3         @str VARCHAR(100)
 4     )
 5     RETURNS VARCHAR(100)
 6     AS
 7     BEGIN 
 8         DECLARE @IncCharLoc SMALLINT
 9         SET @str = LOWER(@str)
10         SET @IncCharLoc = PATINDEX('%[^0-9a-z] %',@str)
11         
12         WHILE @IncCharLoc &gt; 0
13         BEGIN
14          SET @str = STUFF(@str,@IncCharLoc,1,'')
15          SET @IncCharLoc = PATINDEX('%[^0-9a-z] %',@str)
16         END
17         
18         SET @str = REPLACE(@str,' ','-')
19         RETURN @str
20         END 
21     GO

First of all I need to leave spaces so they can be replaced with hyphens, and secondly I only require lower case characters. You may also notice that I'm doing a case transformation against the incoming string. This is due to the SQL database being setup with case insensitivity.

I would like to thank Pinal Dave for his original code.

Software Architecture Evolution Over Time.

Works by Parnas and Shaw set the stage for the development of architectural design patterns that enabled later computer scientists to develop both architectural and module level design patterns. This has empowered architects and developers with the ability to build highly cohesive, loosely coupled large-scale systems.

1. Introduction

The development of codified software design patterns over the last 30 years has been of significance to the software industry. With each new design pattern a new level of abstraction has been developed to simplify or iron out complexity in the systems in where those patterns have been implemented.

However in order to understand the overall architecture of a system we first need to be able to decompose the system into subsystems or modules, with each subsystem or module adhering to a design pattern in order to maintain some sort of maintainability and testability.

2. Discussion

In his 1972 paper, "On the Criteria To Be Used in Decomposing Systems into Modules", Parans discusses the decomposition of software into modules, producing two separate modularisations.

The first is along the lines of functional/procedural responsibility or steps, referred to as the "flowchart" method [1]. The second is along the lines of "Information Hiding".

Parans says of the second modularisation: "Every module in the second decomposition is characterized by its knowledge of a design decision which it hides from all others. Its interface or definition was chosen to reveal as little as possible about its inner workings." [1]

These two sentences are of great interest, as this was the first time the idea of Information Hiding was introduced to the world of Software Architecture and Design.

This idea, sometimes referred to as the Black Box Principal, is of paramount importance to software design, as it is the founding basis for a number of Object Oriented design principals including Encapsulation and the practice of designing to Interfaces rather than concrete classes.

The Black Box Principal allows for a module or class to built in such a way that only its input and output are known, while hiding its internal workings, implementation, or design decisions from consuming applications, modules or classes.

This allows for the design and creation of loosely coupled software systems, which in turn increase testability and reduce the chance of failure due to cascading errors in the event of changes to the module or classes internal design or implementation.

This principal, is in fact an abstraction of the inner workings given module or class in order to implement it’s output in a larger system.

In her 1989 paper "Larger Scale Systems Require Higher-Level Abstractions" Shaw makes the argument for the need of codified high-level design patterns that can be used to make abstracted design decisions at the system and subsystem levels, in order to better understand and produce large-scale systems [2].

In section "3. Composition of Systems from Subsystems", Shaw makes the observation that large systems are constructed by combining subsystems, and that these subsystems have their own internal structures, which could be designed at the system level, rather than the module or class level [2].

This allows for the implementation of higher-level design patterns for specific subsystems that are best suited to their function or responsibilities

However Shaw makes the conclusion that the at the time of publication (1989) Software Architecture was not yet mature enough to develop or support such high-level patterns [2].

As we can see, both Parans and Shaw’s work address software design from a structural, decomposition standpoint, while this may have been the founding for a lot of today’s current structural architecture level design patterns, it fails to take into account the other areas software design that are important in todays object oriented world, specifically those of object creation, structure and behaviour in complex software systems.

Neither of the papers takes into account the inherent complexity of trying to integrate multiple modules to form complex systems, while maintaining portability, testability and high maintainability.

Thankfully, some of these issues have been addressed through the creation of modern architectural and module / class level design patterns.

3. Critical Analysis

Shaw’s paper is a spiritual continuation of Parnas’s earlier "Information Hiding" (Black Box) work, in that Shaw is in fact advocating for the abstraction of lower level modules functions behind a subsystem design pattern, effectively extending the Black Box Principal beyond that of a single module or class to an entire subsystem. This continuation is evident in section 2 of Shaw’s paper where the Software Architecture Level of Design is discussed. Shaw states: "This level of system organization and design is the software architecture level." [2]. Parnas, in his work, did not mention, or perhaps failed to anticipate that large-scale systems may require a number of levels of abstraction in order to effectively produce the system. Shaw’s work rectifies this by explicitly advocating the need for architecture level design.

Architecture level design, is of course a major factor that must be considered when developing systems for modern systems, the choice of architecture level design patterns is of paramount importance when developing large scale systems, as the choice of the wrong design pattern could have catastrophic consequences, for example the Model View Controller pattern, developed at XEROX PRACE in 1978-79 [3] may provide the foundation for a Graphical User Interface (GUI) based system such as a Word Processor, but would be ill suited to a Telephone Exchange System or a Control and Command System.

While Shaw addressed the higher-level concepts of System and subsystem architecture, she failed to address the issues related to lower level modules and classes, thankfully a large number of these issues have been addressed through the acceptance of the patterns and practices published throughout the early 90’s and early 2000’s. Books such as Code Complete [4] and the Design Patterns: Elements of Reusable Object-Oriented Software" [5] introduced a large number of developers and architects to reusable design patterns that allow for the creation and maintenance and testability of modules or classes in a decoupled, highly cohesive manner.

For example, Robert C. Martin in his book "Agile Software Development: Principals, Patterns and Practices" [5] introduces a set of principals known as SOLID. SOLID is an acronym for Single Responsibility, Open-Closed, Liskov Substitution, Interface Segregation and Dependency Inversion. These principals extend on Parnas’s information hiding principals, and Shaw's module abstraction to address the issues related to testability and maintainability in complex systems. For example, the Dependency Inversion principal states that high level modules or classes should not depend on low level modules or classes. This principal is implemented via the Dependency Injection pattern, which extremely similar to the Factory Method [4][5]. Dependency Injection allows for decoupling of modules, which in turn makes it easier to unit test modules.

Well-tested, decoupled modules often have higher levels of cohesion, allowing those modules to be more portable, readable and maintainable in the long term.

There is however still a number of open issues from a software architecture standpoint, as the underlying hardware increases in capacity and complexity the need to continually abstract the inner workings to increase design simplicity of each system is a cause for ambiguity, and misunderstanding. Shaw did mention this in her paper, however, it may well result in a time where those making the design decisions no longer understand the underlying rationale for the design pattern that they have chosen to implement. At a lower level this is evident in the uptake of memory managed programming environments, e.g. Java. It could be argued that Java developers are more ignorant of their memory footprint, compared to C developers. There is a real chance that this could happen at a higher level of design as we continually develop new ways to abstract away from the inner workings of our systems.

4. Conclusion

Parans work created the foundations for a large number of design patterns still implemented in today’s moderns software systems. Shaw extended Parans work in a way that allowed for the abstraction of design decisions to higher levels of a software system. This work has since been extended upon by a number of different people, allowing for reusable software design patterns that give us the ability to build highly cohesive, decoupled systems that are easily tested.

However we may face a time in the future where the continued abstraction of the design decisions leads to an ignorance of the base implementation of a system.

5. References

[1] David L. Parnas. "On the Criteria to be Used on Decomposing Systems into Modules," Communications of the ACM, 15(12):1053-1058, 1972.

[2] Mary Shaw. "Larger Scale Systems Require Higher-Level Abstractions," Proceedings of the Fifth International Workshop on Software Specifications and Design, published 1989

[3] Trygve Reenskaug. ∑"MVC XEROX PARC 1978-79″, http://heim.ifi.uio.no/~trygver/themes/mvc/mvc-index.html

[4] Steve McConnell. "Code Complete" Microsoft Press. 1994

[5] E. Gamma, R. Helm, R. Johnson & J. Vlissides. "Design Patterns: Elements of Reusable Object-Oriented Software", 1994

[6] Robert C. Martin. "Agile Software Development: Principles, Patterns, and Practices " 2002