Category Archives: Development

An Nginx Configuration For Jenkins

Lots of people have posted configurations of Nginx to allow effective proxying of Jenkins when they are both on the same server, but for some reason, it seems that having them on different servers doesn’t seem as commonly discussed. I am using Nginx in my SOHO network to front a few virtual servers, and provide them all via the few IPs I have on my Comcast Business Class connection. That means having a proxy that can serve up the various systems supporting various domains.

We’ve covered how to build a Jenkins server, so for the sake of documenting this additional capability, here’s my configuration:

server {
  listen 80;
  server_name jenkins.domain.com;

  access_log /var/log/nginx/jenkins_access.log main buffer=32k;
  error_log /var/log/nginx/jenkins_error.log;

  rewrite /jenkins/(.*) /$1 last;

  location / {
    proxy_pass       http://192.168.1.115:8080/jenkins/;
    proxy_redirect   off;
    proxy_set_header Host $http_host;
    proxy_set_header X-Real-IP $remote_addr;
    proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;

    # max upload size
    client_max_body_size       20m;
    client_body_buffer_size    128k;
    proxy_connect_timeout      90;
    proxy_send_timeout         90;
    proxy_read_timeout         90;
    proxy_buffer_size          4k;
    proxy_buffers              4 32k;
    proxy_busy_buffers_size    64k;
    proxy_temp_file_write_size 64k;
  }

}

Continuous Integration With Jenkins On Ubuntu 11.10

First, install Ubuntu Server 11.10. Obviously, settings will vary from machine to machine, but when you get to the page for selecting software to be installed, make sure you select both the OpenSSH server and the Tomcat server.

Ubuntu Software Selections

With a fresh server install, you’ll want to assign a static IP to your server. Ubuntu Server 11.10 will likely detect your network card, and set it up during install to use DHCP. But, it makes more sense for a server to have a stable IP. You can change this in /etc/network/interfaces. Change the section that likely reads as:

iface eth0 inet dhcp

to something like:

iface eth0 inet static
  address 192.168.x.x
  netmask 255.255.255.0
  gateway 192.168.x.1

Of course, use whatever local LAN network addresses make sense for you. Either restart the network service (sudo /etc/init.d/networking start) or reboot.

When you’ve rebooted, make sure to update Ubuntu itself.

$ sudo apt-get update
$ sudo apt-get upgrade
$ sudo reboot

Jenkins is a Java app that needs some environment to run it. We’ve already installed Tomcat for this through the Ubuntu installer. You can verify it is running by surfing to: http://%5Byour IP address]:8080. You may also want to configure http://%5Byour IP address]:8080/manager/html. Surfing over to that page will give you the info needed to configure the status page viewer when you fail login on the attempt on the new Tomcat server. The other reason is that this management page allows you to easily deploy the Jenkins WAR too. Download the WAR for the Ubuntu distribution and upload it via the Tomcat manager app.

If you now surf over to http://%5Byour IP address]:8080/jenkins, you will see Jenkins, but in an error state. It will complain that it is “Unable to create the home directory ‘/usr/share/tomcat6/.jenkins’. This is most likely a permission problem.”. Well, at least Jenkins is running! The easy way to solve this is to let Tomcat have access to that folder.

$ cd /usr/share/tomcat6
$ sudo mkdir .jenkins
$ sudo chown tomcat6:nogroup .jenkins
$ sudo /etc/init.d/tomcat6 restart

That should get you going on your adventure in continuous integration with Jenkins.

Concatenate Aggregate Function For SQL Server

Tired of the fact that, still after many years, there isn’t a convenient and built-in way to concatenate strings in an aggegrating query. Well, wait no more!

First, check if your database environment is setup to allow CLR user-defined functions:

SELECT * FROM sys.configurations WHERE name = 'clr enabled'

If value_in_use = 1, you are setup for it. If not, you can turn it on yourself via:

sp_configure 'clr enabled', 1;
GO
RECONFIGURE;
GO

Now, fire up VS2010. Create a Visual C# SQL CLR Database Project from the SQL Server database templates. Right-click on the project in the solution and click on “Add New Item”. From the list, select “Aggregate” (and note all the other goodies you could create). I called the solution/project `ConcatMaxNullable`.

There is plenty of good information on what the various methods stubs are doing. Also, there are some hard-to-find example of concatenate aggregates out there too. Unfortunately, none of them fit my needs. I need one that:

  • Concatenated strings. (Duh!)
  • Could output to the new nvarchar(max) type and not be limited by the 8000 chars of the old nvarchar().
  • Allowed skipping nulls, or replace them with empty strings still delimited.

So, I rolled my own. I’m going to go ahead and simply post my own code for others to use. Should be self-explanatory as the methods don’t really do anything particularly novel.

using System;
using System.Data;
using System.Data.SqlClient;
using System.Data.SqlTypes;
using Microsoft.SqlServer.Server;
using System.Text;

[Serializable]
[Microsoft.SqlServer.Server.SqlUserDefinedAggregate(
  Format.UserDefined,
  IsInvariantToOrder = false,
  IsInvariantToNulls = true,
  IsInvariantToDuplicates = false,
  MaxByteSize = -1
  )]
public struct ConcatMaxNullable : IBinarySerialize
{
  private StringBuilder _accumulator;
  private string _delimiter;

  public Boolean IsNull { get; private set; }

  public void Init()
  {
    _accumulator = new StringBuilder();
    _delimiter = string.Empty;
    this.IsNull = true;
  }

  public void Accumulate(SqlChars item, SqlString delimiter, SqlBoolean skipNulls)
  {
    // if value is null, return if skipNulls is true
    if (item.IsNull && skipNulls.IsTrue)
      return;

    // if we have an actual delimiter
    if ((!delimiter.IsNull) && (delimiter.Value.Length > 0))
    {
      _delimiter = delimiter.Value;
      // accumulate delimiter if we have something already
      if (_accumulator.Length > 0)
        _accumulator.Append(delimiter.Value);
    }

    // if value is null, just add delimiter (above) and return
    if (item.IsNull)
      return;
    else
    {
      _accumulator.Append(item.Value);
      this.IsNull = false;
    }

  }

  public void Merge(ConcatMaxNullable Group)
  {

    if (_accumulator.Length > 0 && Group._accumulator.Length > 0)
      _accumulator.Append(_delimiter);

    _accumulator.Append(Group._accumulator.ToString());

  }

  public SqlChars Terminate()
  {

    return new SqlChars(_accumulator.ToString());

  }

  void IBinarySerialize.Read(System.IO.BinaryReader r)
  {

    _delimiter = r.ReadString();
    _accumulator = new StringBuilder(r.ReadString());

    if (_accumulator.Length != 0)
      this.IsNull = false;

  }

  void IBinarySerialize.Write(System.IO.BinaryWriter w)
  {

    w.Write(_delimiter);
    w.Write(_accumulator.ToString());

  }

}

Once you pasted this in, compile and then get the DLL somewhere the server can see it. Installing it is as easy as running these two little scripts:

CREATE ASSEMBLY ConcatMaxNullable FROM 'C:\\ConcatMaxNullable.dll'
WITH PERMISSION_SET=SAFE
GO

CREATE AGGREGATE [dbo].[ConcatMaxNullable]
(@item [nvarchar](max), @delimiter [nvarchar](8), @skipNulls bit)
RETURNS[nvarchar](max)
EXTERNAL NAME [ConcatMaxNullable].[ConcatMaxNullable]
GO

Then, you can easily do something like this query, which shows you your foreign keys, and the columns involved, in a comma-separated format ready for scripting.

SELECT
  c.[TABLE_NAME], k.[CONSTRAINT_NAME], dbo.ConcatMaxNullable([COLUMN_NAME],',',1) AS [Cols] 
FROM
  INFORMATION_SCHEMA.TABLE_CONSTRAINTS c
INNER JOIN
  INFORMATION_SCHEMA.KEY_COLUMN_USAGE k ON c.[CONSTRAINT_NAME] = k.[CONSTRAINT_NAME] 
WHERE
  c.[CONSTRAINT_TYPE] = 'FOREIGN KEY'
GROUP BY
  c.[TABLE_NAME], k.[CONSTRAINT_NAME]
ORDER BY
  c.[TABLE_NAME], k.[CONSTRAINT_NAME] 

Enjoy!

Architecting A C# Web-Based Application: General Concepts

So, we’ve loosely tossed around our greenfield web application project in our head, and we’ve decided that we’re going to go ahead and develop it. The question then becomes “What’s next?”

At this point, the temptation is to just jump in and start coding, especially if the project is personal (not derived from one or more external stakeholders) and scratches a big, immediate itch. In my case, I have been trying to find a good PM solution that doesn’t burden me if umpteen keypresses to log tickets and has a real ability to workflow items. Many do not fit the bill, and my workload isn’t getting any leaner.

But, instead of jumping right in, we’ll pause for just a little while, and gather up some key concepts we want to put in play in this project. There are a lot of choices to make when you develop a web application. There are concerns over technologies (ex: databasing both relational or KV store, code repositories, etc.), libraries (ex: ORMs, serializers, etc.), platforms (ex: ASP.NET MVC 2), frameworks (ex: jQuery, Sharp Architecture, etc.), development methodologies (ex: TDD, BDD), management approaches (ex: Scrum, XP), architecture concepts and patterns (ex: CQRS, DDD), amongst many others. While each one of these items, like CQRS, is often the subject of multiple blog posts, but I will attempt to cover the salient items with respect to this project within a couple of posts.

In this project, we will cover:

  • Domain-Driven Design (DDD): Domain-driven design has been around for a decade or two, but has really taken off in the last few years, as more developers start tackling software of growing complexity. DDD is a methodology (as in “a framework that is used to structure, plan, and control the process of developing an information system”) that addresses the broad topic of researching, understanding and then designing the conceptual part of whole systems. DDD makes the developer focus on the core functionality by isolating it into a domain model, separate from other concerns, and also by bringing the developer closer to the business user’s language. It does this through the core concepts of “ubiquitous language” and “bounded contexts”, which we will discuss in a later article. The goal of DDD is to create a set of techniques, concepts, patterns and language that directs you to focus on the domain, on the concepts of the system, rather than on the underlying technologies. Said differently, if you want to produce good software, it is more critical that you understand what an Order is and does logically, rather than decide which NoSQL store du jour to use. The latter is often more exciting to technologists, and is also a case of putting the cart before the horse. DDD will help not only build better software through better focus, but also help us do simple things like give us direction in project structure.
  • Command Query Responsibility Segregation (CQRS): This is actually a simple architectural concept or pattern, rather than a comprehensive, technology-specific framework or methodology. The goal is implied in the name itself. The idea is that commands/actions that tell a system what to do and that change the state of the system are segregated, at least logically, from the act of querying the state of the system. Reads and writes frequently contend in busy systems. CQRS is an approach to mitigate that reality of system design. Again, we’ll explore this in its own article soon.
  • Event-Based Architecture (EDA): The ever-present question in any developer’s mind at work is “Where should this block of code go? How should I organize my code?” We want the end-product of our craft to be easy to write, easy to read and easy to change. These needs are encapsulated by a lot of acronyms: DRY, GRASP, SOLID, etc. In very general terms, the main goal is low coupling (low dependencies between functional sections of code, classes or otherwise) and high cohesion (breaking down code into functional sections that are very focused in purpose). Of course, as you break down your code into focused chunks that are independent of each other, the question becomes how do you get them to work together? In comes messaging. These blocks of code coordinate and affect each other via messages. One block of code raises an event (a message) that it did its thing and changed the system, and then interested listeners pick up the message and do what they were created to do in response.
  • Dependency Injection (DI): When we create blocks of independent code, oftentimes there is the need (or temptation) to have one block use another block directly. And so, we pull a direct reference or link to that code; we new up what we need. In so doing, we have increasing the amount of coupling in the system. DI is a way to reduce this coupling. For example, if we implement an EDA-based system, almost every block of code needs to publish their event messages into a system that can then distribute it out to interested listeners. We don’t want to have that code in multiple places; that breaks DRY. We also don’t want to put that code in its own block, and then have every other block link to it; that ruins low coupling. Instead, we use the DI pattern. This allows us to register the event system, and its various parts, in a container or directory that any other part of the system can see and use. That code doesn’t get repeated and the indirect nature of the link allows looser coupling. So, when one block of code needs an event publisher in our EDA system, it calls for one generically (“Hey I need an object that has a way to let me publish an event into the message bus!”) and gets whatever is registered in the system (“Here’s an concrete object for you to do this. It has the method you need.”). Basically, you let a specific part of the system focus on managing dependencies, instead of the immediate code doing it for itself. That makes it easy to change parts. Is your custom-built publish-subscribe code not robust enough? Well, plug in NServiceBus. Built right, with the blocks of code offering up the same interface to achieve the same functions, you should be able to swap systems out.
  • Aspect-Oriented Programming (AOP): AOP is a programming paradigm, a style of coding. The keyword, aspect, describes a focused functional block of code that has a high amount of reuse throughout a system. Aspects are these blocks that are cross-cutting concerns because they “cut across” (a.k.a. “are used in”) many different blocks of unrelated code in the system. A classic example of an aspect is the need for a logging subsystem in an application to support debugging efforts. In a way, whereas DI is a passive way to allow one block of code to use another, AOP is much more active and/or broad-stroked. AOP prescribes a way to apply a block of code (a.k.a. advice, ex: “run this aspect before the block”) across some or all blocks of code (a.k.a. cut points, ex: “all methods in namespace X.Y.Z”). Aspects are a great way to keep such code in one place for maintainability, but effectively apply it where necessary with low cohesion, since the affected block is effectively oblivious to its presence.

We’ll explore a lot of these concepts in detail in subsequent articles. And, I reserve the right to expand this overview list as I discover more topics that deserve an “executive summary” for those fresh to the series. For example, I have not covered testing, continuous integration, and other more technical items that add to our ability to deliver good software. If you think we should cover anything else, feel free to chime in.

Architecting A C# Web-Based Application: Introduction

I am beginning a series of articles on architecting a “serious business” web-focused application. The raison d’ĂȘtre for this is because I have been unable to find a focused, well-documented sample project that exposes practical architecture and guidelines. The overall goal is to put myself on the line as a guinea pig, journal my thought process at every step, take the abuse and, hopefully, generate some positive discussion on the choices I make along the way. The secondary goal is to provide intermediate developers an example of how to approach a common type of project. As such, the series will not be an exploration of cutting-edge technologies, or of advanced coding techniques.

The sample application will be a project management application that follows the great majority of the basic Scrum principles, allows a development team to better manage their workload, and also maximizes the offload of data entry and organizing to the stakeholders as much as possible. I know that this type of application exists in umpteen forms on the net. Let’s face it, it’s basically the “enterprisey” equivalent of Tetris. However, I selected it for three main reasons:

  • The project is a neither too complex, nor uselessly simple in scope
  • It is a domain that should not be foreign to a readership composed of developers
  • I need a good app that fits the way my team works, rather than the overly generic and bloated PM software out there. While this may seem selfish, it’s actually good! I’ll be eating my own dogfood.

I think many developers don’t enjoy the use of most of the PM software out there because they become responsible for too much maintenance/clerical work. Not many packages out there put as much responsibility for the project in the hands of the stakeholders, and if they do, invariably they charge for more licenses. We’ll start simple, focus on providing a great UI, and won’t involve feature-creep just to gain bullet points on a sales presentation. Ultimately, the code base will be provided at large as an open-source project.

The whole solution will consist of your typical moving parts:

  • A web-based client where the majority of the interaction with the system is done, especially the collaborative parts.
  • A task bar application that allows quick data entry, primarily by developers, for common functions.
  • A middle layer built to handle the above two clients, and open for more. This will be where all the rules, workflows, transformations and other tasks happen. This will also force one to consider how to build the layers.
  • The database layer, obviously used to persist data.

The minimal functionality we will provide for a “v1.0” is:

  • The ability to work with the four general “things” found in Scrum: roles, timeboxes, artifacts and rules.
  • Allow business users to easily log ideas (a.k.a. user stories) and track the status thereof.
  • A robust and customizable workflow system for managing rules that, while not very dynamic, should be open to some amount of customization.
  • Some dashboarding for some simple metrics.
  • While we won’t aim to provide a full interactivity suite, it would be nice to build in a way to have a threaded discussion area for each idea or story, such that devs and users can collaborate and flesh out ideas in a way that doesn’t create an email nightmare.
  • A cross-platform, browser-based user interface with a form-based authentication system.
  • Optionally integrate logins with AD or other LDAP system.
  • A small taskbar application that helps you track what you are currently working on and gives feedback on changes in the app.

As you can see, this is not a simple throw-away project. But, neither is it a highly-complex, enterprise application. I hope that this series can help junior and intermediate .NET developers get a feel for how to approach the design of a web application, elevate my own game through feedback both high and low, and for intermediate to senior developers to collaborate on different approaches effectively by having the constraint of a defined scope in play. (Lots of times, comments on blogs like this run the gamut because some people are thinking of more complex projects than others.)

The next articles will cover the typical questions you (should?) have when you kick of a project:

  • What exactly am I talking about? Let’s spend some time on some diagramming and scoping to understand the larger moving parts. We’ll obviously iterate and refactor to get it right along the way, and so let’s not paralyze ourselves early, but we do need to establish some common language between the participants.
  • What tools am I going to use? For example, what framework will we use to get data in and out of our application? And, why? Another example is the many forms of IoC containers out there. Which one and why?
  • How am I going to structure all this practically? Let’s talk about overarching principles/methodologies that we will choose to apply on this project. Let’s also establish the subprojects in the overall solution from a technical standpoint.

Please, if you have any suggestions or comments, serve them up now! Especially if there’s anything in particular, top-level, that you think should be included. And, I hope you join me actively in subsequent posts.