Visualizing Your Connected Systems and Teams

Posted by Alan Barr on Sat 22 February 2020

Using a graph to visualize coupled software systems

A need came up at work to inventory all the software systems we have that run parts of the business. Unfortunately, the data will go stale in short order but for a moment in time it is nice to see how coupled teams are to each other and how they work often fairly regularly in the same domains. There are various graph databases that can help with this but I find visjs dependable and quick to visualize what I need to see based on generating JSON for the nodes and edges to display.

Starting with a CSV

The CSV I received has various data but the most important are two columns: the domains (communication, documents, tasks) and the teams. Each row represents a difference application that a team maintains. Before I write this program I have a good idea of what kind of data I need to generate.

A Node is an object with an ID and a label. An Edge is an object with a to and a from pointing from the source node to the target node. I will need to read a CSV and I will also need to generate two separate arrays of JSON objects with the nodes and edges.

Reading the CSV

These days I default to Linqpad 6, F#, and the library FSharp.Data to parse data formats. The benefit of this is that it is a limited API and I do not have to think too much about the library API to tweak data I can leverage the language to do this. Pandas for example requires some intimate knowledge of how Pandas works to do the work.

open FSharp.Data
open FSharp.Collections
open System.Text.Json
type Domains = CsvProvider<"C:\\Users\\Alan.Barr\\Downloads\\rprt.csv">

Using Linqpad 6 I import the nuget package into the query window and add the code above.

type Node = { id : string; label : string;}
type Edge = { from: string; ``to``: string}
let domains = new Domains()

Next I am defining record types to represent the two kinds of objects I want to generate.

let distinctDomains = domains.Rows |> Seq.map (fun x -> x.Domain) |> Seq.distinct |> Seq.map (fun x -> {id = Guid.NewGuid().ToString(); label = x}) |> Seq.toList
let distinctTeams = domains.Rows |> Seq.map (fun x -> x.Team) |> Seq.distinct |> Seq.map (fun x -> {id = Guid.NewGuid().ToString(); label = x})  |> Seq.toList
let theNodes = Seq.concat [distinctDomains; distinctTeams; ]  |> Seq.toList
let edges = new ResizeArray<Edge>()
let lessNodes = new ResizeArray<Node>()
let lessEdges = new ResizeArray<Edge>()

The CSV type provider makes it easy to work with a strongly typed model of the data. I can access the rows using .Rows and iterate or apply functions on it. My initial need was to identify the unique domains and teams from the data to define the edges. I specifically chose to cast the Sequence types to a list because Seq is lazily evalutated so using Guid.NewGuid() would generate wrong ids. I needed the ids to be consistent. I use ResizeArray so I can dynamically add items to a list.

for domain in domains.Rows do
    let domainId = distinctDomains |> Seq.filter (fun x -> x.label = domain.Domain) |> Seq.head
    let teamId = distinctTeams |> Seq.filter (fun x -> x.label = domain.Team ) |> Seq.head
    edges.Add({from = domainId.id; ``to`` = teamId.id })

theNodes |> Json.JsonSerializer.Serialize |> Dump
edges |> Json.JsonSerializer.Serialize |> Dump

Finally I loop over the rows of the csv and identify which domain and team is on the row and add the edge record to my list of records.

From then on I pipe the data to the Json serializer so I can get json text out into the Linqpad render window.

The end result is an interactive graph showing teams connected to a domain and the number of applications that fit in that domain.

Below is an example of the shell and some data to render the graph

<!doctype html>
<html>
<head>
  <title>Vis Network | Basic usage</title>
   <script type="text/javascript" src="https://cdnjs.cloudflare.com/ajax/libs/vis/4.21.0/vis.min.js"></script>
   <link rel="stylesheet" href="https://cdnjs.cloudflare.com/ajax/libs/vis/4.21.0/vis.min.css" integrity="sha256-iq5ygGJ7021Pi7H5S+QAUXCPUfaBzfqeplbg/KlEssg=" crossorigin="anonymous" />
  <style type="text/css">
    #mynetwork {
      width: 2000px;
      height: 1000px;
      border: 1px solid lightgray;
    }
  </style>
</head>
<body>
<p>
  Create a simple network with some nodes and edges.
</p>
<div id="mynetwork"></div>
<script type="text/javascript">
  // create an array with nodes
  var nodes = new vis.DataSet(
  [
  {
    "id": "e597cad3-074e-4838-93ee-cc1df6e9d73d",
    "label": "Workload Management"
  },
  {
    "id": "31277d13-36a0-41b5-8d6b-e800ea45a1a7",
    "label": "Team 1"
  },
  {
    "id": "721b1818-9a2a-46a2-8e51-8ebe01bfba49",
    "label": "Team 2"
  },
  {
    "id": "77dca992-5764-4ef5-ae8c-03163779c490",
    "label": "Team 3"
  },
  {
    "id": "8e96bcd2-c017-457f-ac61-45990e567437",
    "label": "Team 4"
  },
  {
    "id": "036d44ca-ec64-4a02-9ed0-ed40d278a0ba",
    "label": "Team 5"
  }
  ]
  );

  // create an array with edges
  var edges = new vis.DataSet(
[
  {
    "from": "e597cad3-074e-4838-93ee-cc1df6e9d73d",
    "to": "036d44ca-ec64-4a02-9ed0-ed40d278a0ba"
  },
  {
    "from": "e597cad3-074e-4838-93ee-cc1df6e9d73d",
    "to": "8e96bcd2-c017-457f-ac61-45990e567437"
  },
  {
    "from": "e597cad3-074e-4838-93ee-cc1df6e9d73d",
    "to": "8e96bcd2-c017-457f-ac61-45990e567437"
  },
  {
    "from": "e597cad3-074e-4838-93ee-cc1df6e9d73d",
    "to": "036d44ca-ec64-4a02-9ed0-ed40d278a0ba"
  },
  {
    "from": "e597cad3-074e-4838-93ee-cc1df6e9d73d",
    "to": "036d44ca-ec64-4a02-9ed0-ed40d278a0ba"
  },
  {
    "from": "e597cad3-074e-4838-93ee-cc1df6e9d73d",
    "to": "036d44ca-ec64-4a02-9ed0-ed40d278a0ba"
  },
  {
    "from": "e597cad3-074e-4838-93ee-cc1df6e9d73d",
    "to": "8e96bcd2-c017-457f-ac61-45990e567437"
  },
  {
    "from": "e597cad3-074e-4838-93ee-cc1df6e9d73d",
    "to": "77dca992-5764-4ef5-ae8c-03163779c490"
  },
  {
    "from": "e597cad3-074e-4838-93ee-cc1df6e9d73d",
    "to": "721b1818-9a2a-46a2-8e51-8ebe01bfba49"
  },
  {
    "from": "e597cad3-074e-4838-93ee-cc1df6e9d73d",
    "to": "31277d13-36a0-41b5-8d6b-e800ea45a1a7"
  },
  {
    "from": "e597cad3-074e-4838-93ee-cc1df6e9d73d",
    "to": "31277d13-36a0-41b5-8d6b-e800ea45a1a7"
  },
  {
    "from": "e597cad3-074e-4838-93ee-cc1df6e9d73d",
    "to": "31277d13-36a0-41b5-8d6b-e800ea45a1a7"
  }
]
  );

  // create a network
  var container = document.getElementById('mynetwork');
  var data = {
    nodes: nodes,
    edges: edges
  };
  var options = {
        nodes: {
            shape: 'circle',
      size: 10
        },
        layout: {    improvedLayout : false  },
    };
  var network = new vis.Network(container, data, options);
</script>


</body>
</html>