Developer Diary: Tales from a deeply-nested JSON object

Recently at work I was excited to design and develop a tool which would improve a workflow for our support team members. This involved taking a CSV and comparing the data within it to versions of that data that existed in our database. Previously, any given team member achieved this by downloading the published data from the database to a CSV file, and manually comparing it to the new file. This process could take a long time, and there was always the case for human error.

So, after speaking with SMEs from that team and getting a clear idea of what the ask was, I :

• Reached out to the architects and developers of the code upstream of the problem we were solving for, to get any questions answered and have an open line of dialogue during development.
• Created all of the epics, stories, and other bits in JIRA, pseudo-coded, created wireframes, and outlined every other detail I could foresee in dev documentation. I presented the outline to the team that would be working on the tool with me, and updated my notes based on their feedback, then we started working on the prototype.

Outline of the program:

1. We needed to take in and parse a CSV to JSON.
2. Create a nested object from the flat JSON in created in step 1, by identifying relationships between any given pair of rows. The nesting needs to be parent > child, and follow a pre-defined structure (see code snippet below).
3. POST that object as JSON to an existing endpoint which returns the difference between the provided JSON, and the JSON that is in a database.

Here are some sample rows from the CSV as an example of what we were working with:

Each row is related to another by Parent ID

These files contain information that can be used to relate each row in a hierarchical fashion. The levels of the hierarchy (ex. Fry Cook, Owner, Company), row count in the file, and column headers are all variables that I needed to keep in mind when developing this solution.

With that in mind, let me show you what the JSON being created in step 2 needed to look like before passing it as the body of the POST call:

This structure repeats for each row in the CSV

I wrote the program to be ran as server-side JavaScript using node.js, and I used the csvtojson node module to convert the source CSV file to JSON.

With the flat JSON in hand, I decided that I needed to start creating objects that contained data about each level of the hierarchy:

• Level
• Order in the hierarchy
• Direct child level
• Direct parent level
• Count of rows / nodes at this level

This object was created for each level of the hierarchy

Next, I arranged these objects in an array (objects for this article), such that their index = object.order, thus assuring that I could make assumptions that objects[0] would always reference the root level, and objects[length-1] would reference the lowest leaf level.

So now I had n amount of objects in an array, each containing helpful bits of metadata that would enable me to associate individual objects (previously rows in a CSV) with their parent node, and I could carry on with building my nested object.

Another team member was working on the file validation portion of the program, which would mean that the file being passed to this portion of the program would be validated and vetted. I realized I had a different problem to solve for, I could not guarantee that the rows within the CSV file were in any helpful order. This meant that when I started iterating through the JSON, and trying to access object property values, and nest child nodes within parent-nodes, I would first have to iterate over each level and all of the nodes belonging to that level. This approach of breadth over depth would allow me to guarantee that when we continued iterating at the next level in the hierarchy, all of the nodes at the level would have parents that existed in my object.

At this point, I had a problem, I couldn’t immediately wrap my mind around what the code would look like to get and set objects at dynamic locations, but I knew that I would need to create the keys for my dot notation and concatenate them with counters for each level of the hierarchy. This combination is what I called the accessString in my code, and it was the golden-ticket in developing a solution to the aforementioned problem.

Now I was creating a string like this: sets.0.sets.0.sets.0 where starting from the right, each 0 would be incremented to the maximum of the counter it represented, before rolling back to zero and moving over and incrementing the next counter.

These counters were key to ensuring we could successfully get and set values in our object

I could use the access string, along with a few “tried and true” string and array methods to help populate our object. I employed my Google skills to see if anyone had already solved this problem, and found a pretty elegant solution, here’s what it looked like after I made it suit our needs:

With this in place, I wrote code that identified and pushed nodes to the root-node, at the highest level of the hierarchy, and continued to iterate to identify and push child nodes to their respective parent node, until the counters reached their max.

With all of that, I now had the file I needed to POST to our endpoints or write to a file. After setting up the POST and file write, the prototype was at our pre-defined MVP stage!

A new approach I took while writing this program was prototyping outside of the destination application. I am often writing new code within an existing app, and extending it’s functionality. This time I wrote the new code separately from the app while another team member worked on the front-end within the app. I then taught a SME from the team of stakeholders how to use the prototype, that way we could begin receiving feedback on the core functionality of the tool while we migrated it into our MEAN stack application, and the support team could begin benefiting from the program.

That was a rewarding and productive sprint — but I’m even more excited for when we deliver the full experience!

Until the next episode.

-Cam