http://www.sanjay.io/feed.xmlSanjay Uttam's Bloghttp://www.Sanjay.io/2014/03/unit-tests-taking-it-too-far/http://www.sanjay.io/2014/03/unit-tests-taking-it-too-far/<p>Typically, unit testing your data-access class follows a pretty vanilla pattern. If you're using the repository pattern, for example, it's going to look something like:</p> <pre><code>public interface ICustomerRepository { Customer Get(int id); } public class CustomerRepository: ICustomerRepository { public Customer Get(int id) {} } [TestMethod()] public void Test_Customer_Repository_Returns_Customer_When_Passed_Valid_Id() { //setup var mockICustomerRepo = new Mock&lt;ICustomerRepository&gt;(); var customerRepo = new CustomerRepository(mockICustomerRepo.Object); mockICustomerRepo.Setup(c =&gt; c.Get(1)) .Returns(new Customer{ Id = 1, FirstName = "Sanjay", LastName = "Uttam"}.Verifiable(); var result = customerRepo.Get(1); //assert some stuff } </code></pre> <p>That's all fine and well, and it may sufficient in certain applications. But what happens when you want to test whether your repository methods populate objects correctly? Perhaps you've got one or two queries that are still using stored-procedures for performance or legacy reasons. In that case, the above test is really evaluating the correctness of your Get method in the Customer class on the assumption that at run-time, the stored-procedure or query returns the correct data. </p> Wed, 05 Mar 2014 05:00:00 Z2014-03-05T05:00:00Z<p>Typically, unit testing your data-access class follows a pretty vanilla pattern. If you're using the repository pattern, for example, it's going to look something like:</p> <pre><code>public interface ICustomerRepository { Customer Get(int id); } public class CustomerRepository: ICustomerRepository { public Customer Get(int id) {} } [TestMethod()] public void Test_Customer_Repository_Returns_Customer_When_Passed_Valid_Id() { //setup var mockICustomerRepo = new Mock&lt;ICustomerRepository&gt;(); var customerRepo = new CustomerRepository(mockICustomerRepo.Object); mockICustomerRepo.Setup(c =&gt; c.Get(1)) .Returns(new Customer{ Id = 1, FirstName = "Sanjay", LastName = "Uttam"}.Verifiable(); var result = customerRepo.Get(1); //assert some stuff } </code></pre> <p>That's all fine and well, and it may sufficient in certain applications. But what happens when you want to test whether your repository methods populate objects correctly? Perhaps you've got one or two queries that are still using stored-procedures for performance or legacy reasons. In that case, the above test is really evaluating the correctness of your Get method in the Customer class on the assumption that at run-time, the stored-procedure or query returns the correct data. </p> <!--excerpt--> <p>A simple contrived example:</p> <pre><code>//Customer Class public class Customer{ public int Id {get; set;} public string FirstName {get; set;} public string LastName {get; set;} public DateTime DateOfBirth {get; set;} } //Your DB query or stored-procedure contents SELECT Id, FirstName, LastName FROM Customer WHERE Id = @Id </code></pre> <p>In this example, our query is missing the DateOfBirth column. That's an easy omission to make, especially if the column is new in the Customer table. There is at least one major issue with this besides Customer.DateOfBirth not being populated. Your unit test doesn't actually verify that the correct data is returned, because the unit test returns an object that hasn't been created from the database engine. It's something easy to fix in this example, but I've worked on systems in the past where the most critical logic in the application was a stored procedure that pulled logs of what a user has consumed (including quantity, calories, serving size, etc.). The stored-procedure was highly efficient when compared to working with an ORM, but at the cost of complexity. It did all sorts of fun stuff, including PIVOTs, CTEs, and running some basic calculations. It's easy to miss making changes in that type of scenario, but you can work around that.</p> <p>One approach to this is using an in-memory database at test-time to actually run the queries you need to run and ensure that it does in-fact provide the desired result. Here is the basic sequence of steps we need to take:</p> <p>1 - In our unit-test class' set-up event, create a SqlLite instance, create all objects, and insert all relevant data</p> <p>2 - Run the repository class methods you need to test. You'll need to create a mechanism for injecting a database engine instance into your actual repository class, so you can provide an instance of SqlLite rather than SqlServer (or whatever you're using at runtime). </p> <p>3 - In your unit test class' tear down event, destroy the database you created.</p> <p>There is a full link to the source below, but here's a snippet of the important bits.</p> <pre><code> [TestClass] public class CustomerRepositoryTests { public Container Container { get; set; } public IDbConnectionFactory DbFactory { get; set; } [TestInitialize] public void TestInitialize() { Container = new Container(); Container.Register&lt;IDbConnectionFactory&gt;( new OrmLiteConnectionFactory(":memory:", false, SqliteDialect.Provider)); DbFactory = Container.Resolve&lt;IDbConnectionFactory&gt;(); using (var db = DbFactory.Open()) { db.CreateTable&lt;Customer&gt;(overwrite: true); db.Insert(new Customer { FirstName = "Sanjay", LastName = "Uttam" }); } } [TestMethod] public void Test_Get_Customer_By_Id() { var result = new CustomerRepository(DbFactory).Get(1); Assert.AreEqual(1, result.Id); } } </code></pre> <p>Not surprisingly, this might be overkill for you, but hey, as with anything else, it depends. There are a couple of pros and cons that come to mind immediately. </p> <p>You can find a <a href="https://github.com/sanjayuttam/Sanjay.Io.UnitTestingRepositories">demo</a> on GitHub. Note, my demo uses ServiceStack 3.9.71, but this approach doesn't require you to.</p> <h2>Pros</h2> <ul> <li><p>Opportunity to test actual data-access logic. This is likely only equitable in terms of time/value when data-access code is complex and/or external (e.g., stored procedure) or when there data-specific conditions that you must verify are treated correctly.</p></li> <li><p>Ability to create certain data-related conditions and ensure data-access acts in the desired manner</p></li> </ul> <h2>Cons</h2> <ul> <li><p>Objects and data must be accurately created at set-up and tear-down (There might be a way to create the in-memory database on the fly with some hacking, though)</p></li> <li><p>SqlLite query syntax will most likely not be a 1:1 with your actual database engine. </p></li> </ul> http://www.Sanjay.io/2014/01/the-hackers-guide-to-linkedin/http://www.sanjay.io/2014/01/the-hackers-guide-to-linkedin/<p>Most of the time when I'm talking to other technology professionals about LinkedIn, they get pissed off and go into some rant related to LinkedIn being primarily a communication channel for receiving recruiter spam so let's get this out of the way first. You're going to get spammed at least a little. But...</p> Wed, 22 Jan 2014 05:00:00 Z2014-01-22T05:00:00Z<p>Most of the time when I'm talking to other technology professionals about LinkedIn, they get pissed off and go into some rant related to LinkedIn being primarily a communication channel for receiving recruiter spam so let's get this out of the way first. You're going to get spammed at least a little. But...</p> <!--excerpt--> <ul> <li>You can probably spot these kinds of messages from a mile away</li> <li>You don't have to respond to these!</li> <li>Are you the only person in the world that is not getting spam from every other direction already?</li> <li>These tips will hopefully reduce the signal-to-noise ratio if you are already on LinkedIn</li> </ul> <p><strong>Some Background</strong></p> <p>Moving on then, as with most things you are going to get out of it what you put in. The tips below are techniques I have used to optimized for getting individuals to look at my profile, and hopefully connect. Why? Simple really - most of the time when you're in LinkedIn and you're looking at any kind of results, they are ordered by network distance from you (e.g, they are your connections, your connection's connections, etc.). You want to be in other peoples results- that means you want to be networked with as many people as you can, and it's better for everyone if they are in related industries and in your general physical location. This way, you'll be in more results for more searches that are being conducted. That brings to me to next point. This is most certainly over simplified, but there are a few different profiles of individuals that are recruiting for jobs.</p> <ul> <li><p>Third-Party Agencies: When people think of recruiters as annoying, they're probably thinking of these guys. Third-party recruiters make a cut off your placement (how it's structured varies depending on the company, role, and other factors). This doesn't make them evil, though, there are plenty of third-party recruiters that are trying to fill roles that are cool. Some of them just suck at it, but doesn't that apply to any profession really?</p></li> <li><p>In-house Recruiters: These recruiters work for the company that they are trying to fill a job at. You're probably a lot less likely to get spammed from these guys since they likely have multiple channels of getting candidates in their pipeline. These other channels might be public career websites that, over time, collect thousands of resumes that they can draw on at any time. Guess what else? These guys frequently use third-party recruiters as a time saver, or simply to augment staff temporarily (remember what I said about third-party recruiters having roles that are cool?).</p></li> <li><p>Hiring Managers: Sometimes hiring managers poke around LinkedIn themselves. This isn't that rare, especially when it comes to small to mid-size companies. I've definitely been contacted by multiple CTOs via LinkedIn.</p></li> </ul> <p>So why should you care about any of that? You can use LinkedIn to your advantage if you do it right. </p> <p>Tips</p> <p><strong>Have a Reasonably Complete Profile</strong></p> <p>If you've got a profile that literally gives no information other than the fact that you are a developer, you better believe you're going to get spammed <em>more</em> by recruiters that are working for recruiting agencies. Your profile doesn't allude to any specific skills, and for them the amount of work to copy and paste some garbage into an InMail really has zero downside - it's basically skills-phishing. </p> <p>Also, have a photo. It is preferable to have a real one, but you want something just so you are more memorable and recognizable in search results.</p> <p><strong>Create an e-mail address specifically for LinkedIn &amp; Put it Somewhere Visible in Your Profile</strong></p> <p>I recommend something like FirstName.LastName.LinkedIn@Gmail.com...this does a few things for you: - Whenever you get mail that is not LinkedIn related to this address, you know that it's spam. Add it to your spam filter, and that's the end of that. - By default, LinkedIn doesn't let users that aren't connected to you message you (you might get a few free InMails or something but it's very limited if you're a free user). By providing an e-mail address, we're giving people a way to contact us if they want, or connect with us. I've been doing this since around 2006 - I don't have any kind of spam issues (I use GMail). We want to encourage individuals to connect with us, and we want to make that easy for them.</p> <p><strong>Allow others to see when you view their profile</strong></p> <p>This might seem counter-intuitive at first but it's actually pretty simple reasoning. We want other people to see when you view their profile because that will hopefully prompt them to view your profile, and...you guessed it, connect with us! This is all about surface area. But that isn't' even the best part. You can use this to your advantage based on who you visit. Let's say I have a completed profile. Let's also say that I am really interested in a job at Microsoft's NYC Office. I should conduct searches and find <em>technical recruiters at Microsoft NYC</em>. They are going to check who has viewed their profile. That will give them at least an opportunity to click on your profile. Hopefully, your profile looks interesting enough for them do one of the following:</p> <ul> <li>Connect with you</li> <li>Connect with you and message you about an opportunity</li> <li>E-mail you at the address we created specifically for LinkedIn, since it is on your profile</li> </ul> <p>Even connecting with you is great, because we're now that much more likely to be in the result list for their searches, and searches of the individuals they are connected with.</p> <p>This setting can be changed by going to: Clicking on the top right icon of your image (if you've got one) > Privacy and Settings > Select what others see when you've viewed their profile</p> <p><strong>Update Frequently </strong></p> <p>Update your profile as you think of new skills, complete more projects, and certainly get a new job. While I could not empirically explain the importance of this, some non-trivial portion of the results ranking is based on the last time you have updated your profile. Try it for yourself, update something minor on your profile everyday this week and see if you get more messages (You're going to want a reasonably completed profile, of course).</p> http://www.Sanjay.io/2013/12/everything-that-is-wrong-with-on-demand-loading/http://www.sanjay.io/2013/12/everything-that-is-wrong-with-on-demand-loading/<p><figure> <a href='https://twitter.com/BoredElonMusk/status/412994747632857089/photo/1' target='_blank'> <img src='/images/VennDiag.PNG' alt='You're an idiot Venn Diagram' class='image image-centered'/> </a> </figure></p> <p>I think at some point before AJAX was the norm for consumer-facing web-apps, I probably thought on-demand loading was cool, too. Now, it's usually a usability nightmare. It's even worse if we're talking about responsive sites on mobile devices because they've got to serve a variety of clients on a variety of connection speeds. Last night, while drinking a glass of wine and trying to do some on-line shopping on my phone, I ended up at multiple e-commerce websites that had implemented on-demand loading. None of them nailed it. Some were better than others. The exact use case on all the sites was this:</p> <ol class='style1'> <li>User selects some category ("Men's Shirts" or "sale")</li> <li>A list of the first N results is loaded, the user can scrolls vertically (e.g., 1 column, N rows) </li> <li>When the user gets to the bottom of the results, either the page ajax-loads the next N number, or the user can click to initiate the loading of the next N results.</li> </ol> Mon, 16 Dec 2013 05:00:00 Z2013-12-16T05:00:00Z<p><figure> <a href='https://twitter.com/BoredElonMusk/status/412994747632857089/photo/1' target='_blank'> <img src='http://www.Sanjay.io/images/VennDiag.PNG' alt='You're an idiot Venn Diagram' class='image image-centered'/> </a> </figure></p> <p>I think at some point before AJAX was the norm for consumer-facing web-apps, I probably thought on-demand loading was cool, too. Now, it's usually a usability nightmare. It's even worse if we're talking about responsive sites on mobile devices because they've got to serve a variety of clients on a variety of connection speeds. Last night, while drinking a glass of wine and trying to do some on-line shopping on my phone, I ended up at multiple e-commerce websites that had implemented on-demand loading. None of them nailed it. Some were better than others. The exact use case on all the sites was this:</p> <ol class='style1'> <li>User selects some category ("Men's Shirts" or "sale")</li> <li>A list of the first N results is loaded, the user can scrolls vertically (e.g., 1 column, N rows) </li> <li>When the user gets to the bottom of the results, either the page ajax-loads the next N number, or the user can click to initiate the loading of the next N results.</li> </ol> <!--excerpt--> <p><span class="byline">How does this suck? Let me count the ways.</span></p> <ol class="style1"> <li>You just trashed my back button. I'm scrolling down your seemingly infinite list of shirts. I find one that I want to take a closer look at and in doing so. I click on it, decide I either do or don't want to add it to my cart. I'd like to continue shopping now, but when I hit back, I'm at the "top" of your list (e.g., I need to scroll down and on-demand load to get to where I left off). Navigating back to the product list doesn't help of course. That's a pretty terrible experience. I am now forced to open in a new tab when I return to your product list. (Hint: Don't break this functionality if you're doing on demand loading coughhugoboss.comcough)</li> <!-- <img src="http://www.Sanjay.io/images/BBBrokenBackButton2.gif" class="image image-left"> --> <li>In using on demand loading, you are relinquishing the control you had over the quality of my experience when I visit your site. That's because although I'm staring at your fancy-loading-animated-gif-that-looks-eerily-like-facebooks, it can only go as fast as my connection permits. There were times last night when I was browsing and sites would take between 5 and 10 seconds to load the next N results. (Hint: If your Ajax request takes as long or longer than a full page load, that's a problem). Of course, this could be due to multiple factors -on either end- but as far as I'm concerned you need to address as many of them as you can from a developer standpoint:</li> <ul class="style1"> <li> My connection was slow, or inconsistent (fallback to normal pagination on your end if speed is below X/s or at least provide a button prompting me to load the next page) </li> <li> Your server is not [consistently] fast (hook up a reverse proxy, data-cache...reduce payload, etc.) </li> </ul> <figure> <img src='http://www.Sanjay.io/images/BBSlowLoading.png' alt='Brooks Brothers Loads Slowly' class='image image-centered'/> <figcaption>Going to Brooks Brothers on a slowish connection? <br/> You're going to see a lot of this.</figcaption> </figure> <br/> The bottom line is, don't punish me for having a less-than-perfect connection if you can help it especially when you are assuming your site will be consumed by mobile clients (e.g., it's responsive). <li>This one is somewhat related to the speed issue above. If you're going to do on-demand loading for this type of scenario, provide the user with a way to initiate the "next page" themselves. This provides the user with some level of understanding as to the proposed next step in the process. If my connection is slow, then at least I know there is in fact another "page" to see. If you don't provide this button, I really have no choice but to scroll vertically to the bottom of the page, and cross-my fingers to see if there are more products. If things don't start happening pretty quickly, I may very well assume I have seen all there is to see.</li> <!-- <img src="http://www.Sanjay.io/images/BossPromptLoad.gif" class="image image-left"> --> <figure> <img src='http://www.Sanjay.io/images/BossClickToLoad.png' alt='Brooks Brothers Loads Slowly' class='image image-centered'/> <figcaption>Boss got it right. <br/> The user prompts the next page load.</figcaption> </figure> <li>You pretty much made it impossible to find anything the second time in a time efficient manner, unless I memorized the <em>exact name of the product</em> (hint: I didn't, I'm drinking a glass of wine for cryin' out loud.) Why is this? Well since you're doing this fancy on-demand loading you somehow decided to just pretend that even though it's actually just paging, I don't deserve to ever know what page I'm on. If this was 2003, I probably would have just made a mental note of the page that had that shirt that I liked on it. That isn't even a possibility given this user-experience. This brings me to the next point...</li> <!-- <img src="http://www.Sanjay.io/images/BBBrokenBackButton2.gif" class="image image-left"> --> <li>You have totally trashed any kind of <em>find</em> functionality. I know what you're thinking, who uses <em>find</em> when they're using their mobile device? I get that, and I agree. But just because it's only the 20% that are going to try to use this, does that give you the right totally break it? As a user, I'd like my find button to work, please.</li> </ol> <p>So, if you're thinking about using on-demand loading, think through it a little bit. Make the end-user experience the most important factor you consider. Don't assume they're on a fast connection, especially if you're going the responsive route. My patience on a mobile device is not the same as it is when I've got a keyboard and mouse in front of me.</p> <p></rant></p> http://www.Sanjay.io/2013/12/checking-if-youre-being-filtered-on-outbound-ports/http://www.sanjay.io/2013/12/checking-if-youre-being-filtered-on-outbound-ports/<p>Earlier this week I spent some time playing with Redis on my local dev box. Set-up was smooth sailing; install virtual box w/Ubuntu, Redis, set up port forwarding from a host port to a guest port, grab ServiceStack.Redis...I was on my way. Things got a bit more interesting after I deployed Redis to Azure. I changed the configuration in my little console test application to point to the Azure instance. It was running on the default Redis port, 6379. Fired up the console application...tried a SET. Timed out. Tried it again - same result. </p> Fri, 13 Dec 2013 05:00:00 Z2013-12-13T05:00:00Z<p>Earlier this week I spent some time playing with Redis on my local dev box. Set-up was smooth sailing; install virtual box w/Ubuntu, Redis, set up port forwarding from a host port to a guest port, grab ServiceStack.Redis...I was on my way. Things got a bit more interesting after I deployed Redis to Azure. I changed the configuration in my little console test application to point to the Azure instance. It was running on the default Redis port, 6379. Fired up the console application...tried a SET. Timed out. Tried it again - same result. </p> <!--excerpt--> <p>After verifying that the service was running on the box over SSH, I concluded it had to be some outbound filtering. Sure, I could fire up telnet and go to 6379 on my Redis box, but that assumes that there isn't some Azure configuration blocking that port, IPTables isn't blocking that port, and there isn't some Redis configuration I was missing. Those were all important, but I <em>only</em> wanted to verify I was able to get outbound on 6379. Turns out...there is a tool for that: <a target="_blank" href="http://portquiz.net/">PortQuiz</a></p> <p>It's super simple. Basically, the portQuiz server is listening on all ports. This means you can go ahead and telnet to any port. If it times out, you're being filtered outbound. This way, you can be sure it's actually port filtering and not a missing or bad configuration somewhere else in your stack. </p> <p>So, check it out next time you want to eliminate this as a possibility, <a target="_blank" href="http://portquiz.net/">PortQuiz.net</a></p> http://www.Sanjay.io/2013/12/executor-process-exited-or-exception-has-been-thrown-by-the-target-of-an-invocation/http://www.sanjay.io/2013/12/executor-process-exited-or-exception-has-been-thrown-by-the-target-of-an-invocation/<p>I came across this error today while moving some source around TFS. I had moved a project from from TFS Project Collection to another, done a get latest and everything built just fine. However, when trying to run unit tests after all the tests were discovered, I saw the "Exception has Been Thrown by the Target of an Invocation When Attempting to Run Unit Tests" error message in the Output window. I figured I'd try doing a get latest and running the tests on my other machine. That worked without issue. That was the first big clue; this had to do with my VS, not the source that was checked in. After trying pretty much everything in this <a href="http://hjerpbakk.com/blog/2011/5/30/exception-has-been-thrown-by-the-target-of-an-invocation.html" target="_blank">blog post</a>, I did a quick compare on VS versions; noticed the machine where everything was working had Visual Studio Update 3, the machine that was throwing this error did not. I went ahead and installed that. Great, that had to be it right? WRONG. That just caused me to get a different error, even <em>more</em> nebulous error; "Executor Process Exited". Clearly this was still environmental. Turns out the answer was going into Test Settings and changing the default processor architecture to x64. That is, Test > Test Settings > Default Processor Architecture > x64. Cue unicorns and rainbows.</p> Tue, 03 Dec 2013 05:00:00 Z2013-12-03T05:00:00Z<p>I came across this error today while moving some source around TFS. I had moved a project from from TFS Project Collection to another, done a get latest and everything built just fine. However, when trying to run unit tests after all the tests were discovered, I saw the "Exception has Been Thrown by the Target of an Invocation When Attempting to Run Unit Tests" error message in the Output window. I figured I'd try doing a get latest and running the tests on my other machine. That worked without issue. That was the first big clue; this had to do with my VS, not the source that was checked in. After trying pretty much everything in this <a href="http://hjerpbakk.com/blog/2011/5/30/exception-has-been-thrown-by-the-target-of-an-invocation.html" target="_blank">blog post</a>, I did a quick compare on VS versions; noticed the machine where everything was working had Visual Studio Update 3, the machine that was throwing this error did not. I went ahead and installed that. Great, that had to be it right? WRONG. That just caused me to get a different error, even <em>more</em> nebulous error; "Executor Process Exited". Clearly this was still environmental. Turns out the answer was going into Test Settings and changing the default processor architecture to x64. That is, Test > Test Settings > Default Processor Architecture > x64. Cue unicorns and rainbows.</p> http://www.Sanjay.io/2013/11/graphconnect-nyc/http://www.sanjay.io/2013/11/graphconnect-nyc/<p>Last week I attended the NYC Neo4j Meetup on Data Modeling &amp; Natural Language Search, even better, I happened to win tickets to GraphConnect NYC which was just a few days later. I decided to compile some notes from the event, but rather than just bullet-pointing out the content, I decided to compile a brief synopsis of each of the forums I attended so you know which slides to review or video to check out first. This should be a pretty decent place to start if you're just looking to get your bearings when it comes to Graph DBs and Neo4j. There are also a bunch of videos available at <a href="http://watch.neo4j.org/" target="_blank">http://watch.neo4j.org/</a>.</p> Fri, 15 Nov 2013 05:00:00 Z2013-11-15T05:00:00Z<p>Last week I attended the NYC Neo4j Meetup on Data Modeling &amp; Natural Language Search, even better, I happened to win tickets to GraphConnect NYC which was just a few days later. I decided to compile some notes from the event, but rather than just bullet-pointing out the content, I decided to compile a brief synopsis of each of the forums I attended so you know which slides to review or video to check out first. This should be a pretty decent place to start if you're just looking to get your bearings when it comes to Graph DBs and Neo4j. There are also a bunch of videos available at <a href="http://watch.neo4j.org/" target="_blank">http://watch.neo4j.org/</a>.</p> <!--excerpt--> <hr/> <p><span class="byline">New Opportunities for Connected Data</span> <em>Emil Eifrem / <a target="_blank" href="http://watch.neo4j.org/video/69627080">Presentation (Video)</a> / <a target="_blank" href="http://www.slideshare.net/jaxlondon2012/new-opportunities-for-connected-data-ian-robinson">Presentation (Slides)</a> / <a target="_blank" href="https://twitter.com/emileifrem">@emileifrem</a></em></p> <p><strong>Intro/GraphDBs and Data Complexity</strong></p> <p>Adoption rates of graph data-bases continue to increase, with telecom, financial services, and web/ISV-firms driving the bulk of growth. This is primarily due to the evolution of data and data connectedness (Note: coming from a relational database model, you can think of connectedness as a data-model with lots of joins). Although graph databases and graph modeling can seem intimidating, it's actually a very natural way to model the real-world.</p> <p>For example, compare the complexity of answering the below questions in their respective domains in a relational DB versus a graph DB.</p> <ul class="style1"> <li>Subway or Transit Systems: "How do I get from point A to point B?"</li> <li>Data-Centers or Networks: "If switch X goes down, what systems are impacted?"</li> <li>Recommendation Engines: "What did customers who bought this product also buy?"</li> <li>Social Networks: "How many of my friends know person X?"</li> </ul> <p>These can certainly be modeled in a relational world, and the questions answered using appropriate algorithms but all of that extra work becomes unnecessary if you use a graph DB instead. Adoption rates seem to indicate this assessment is correct, especially in products where determining correlations and causality is a key feature. </p> <p><strong>Graph DBs, Features and Benefits</strong></p> <p>In addition to being intuitive and natural to model, graph DBs also provide the advantage of providing constant query times even as the data-set grows. That is, you are only paying the traversal of nodes (compare to outer joining on a large table). Additionally, since graph DBs are have non-rigid structures, it is easier to evolve the structure of a graph DB when compared to a relational DB.</p> <p><strong>Querying a Graph DB</strong></p> <p>A graph consists of nodes (entities) and vertices (lines to other nodes). Each of these vertices can be labeled with a relationship descriptor. Additionally, each node can have N number of properties.</p> <p><a href="http://www.neo4j.org/learn/cypher" target="_blank">Cypher</a> is the language Neo4j provides to describe and query the graph. Cypher queries can provide the engine the following data:</p> <ul class="style1"> <li>A starting node</li> <li>What relationships to traverse</li> <li>What data to return</li> <li>What property values to match</li> <li>How to order the data</li> </ul> <p><span class="byline">Designing and Building a Graph Database Application</span> <em>Ian Robinson / <a target="_blank" href="http://vimeo.com/76710631">Presentation (Video)</a> / <a target="_blank" href="http://iansrobinson.com/">Blog</a> / <a target="_blank" href="https://github.com/iansrobinson">GitHub</a> / <a target="_blank" href="https://twitter.com/iansrobinson">@IansRobinson</a></em></p> <p>Not unlike the relational world, it's critical to model data correctly when building an application that is backed by a graph DB. It's helpful to know the questions you are going to ask of your data when you are building your data-model.</p> <p>Given a system that is used for knowledge management and exploring one's professional social network, a typical question asked of the system might be &#34;Which people who work for the same company have the same skills as I do?&#34;</p> <p>Start by:</p> <ul class="style1"> <li>Identifying entities (People, Companies, Skills)</li> <li>Identifying relationships (A person WORKS_FOR a company, a person HAS_SKILL)</li> </ul> <p>Convert these relationships and entities into cypher, and examine how easy (or difficult) it is to answer your sample question(s). It may also be helpful to create a sample graph and ask the question(s) with the graph model in front of you (See slide 14).</p> <p><strong>Testing</strong></p> <p>Tests should be similar to tests that you'd have in an application with a relational store. In addition, you should be able to test that your data model is sufficient/adequate, as well as be able to ensure the queries you designed return the correct data. Of course, these tests also serve the purpose of documentation and act as regression tests.</p> <p>In Java, there is an entirely in-memory version of Neo4j that should only be used for testing. This engine can be injected via dependency injection (as you would do with other dependencies). In doing so, you should also include a small, static data-set. This should provide you the ability to easily execute your queries with full knowledge of what the correct response should be. The exact strategy (how this gets injected, etc.) will depend on your architecture.</p> <p><strong>Neo4j Application/Platform Choices</strong></p> <p>Neo4j has 3 options as far as how it can be installed/leveraged.</p> <ol class="style1"> <li>Embedded</li> <ul class="style1"> <li>Hosted in a Java process</li> <li>Provides native access to the Neo4j Java API</li> </ul> <li>Server</li> <ul class="style1"> <li>Wraps the embedded instance</li> <li>Provides a JSON REST interface</li> </ul> <li>Server with Extensions</li> <ul class="style1"> <li>Provides a way to execute custom [Java] code on the server</li> <li>Provides you with a hook to control the HTTP request and response format</li> <li>Basically, provides a hook where into the HTTP pipeline (if you request &#34;/get_skills/&#34; handle it like this...)</li> </ul> </ol> <p>There are specific code examples for writing unit tests of embedded code as well as Server Extensions starting at slide 30 of Ian's presentation.</p> <p><span class="byline">How Marvel Comics Uses Graph Databases</span> <em>Peter Olson / <a target="_blank" href="https://twitter.com/iansrobinson">Presentation (Video)</a> / <a target="_blank" href="https://twitter.com/Dethtron5000">@Dethtron5000</a></em></p> <p>Marvel has a significant inventory of stories, sprawling storylines, authors, and characters. Additionally, stories from the marvel Universe don't obey the rules that we are likely used to modeling. For example, characters can have a property for attribute for a very limited time, die and come back to life, exist in multiple places at once...there are basically no rules, whatsoever. This would be a relational modeling nightmare.</p> <p>Graph DBs are being used to solve a number of challenges associated with this universe. For example, given this scenario: &#34;A character is a superhero at a time T, which occurs in issue I. This character is also related to an overarching theme in the Marvel history, and is part of a team.&#34;</p> <p><span class="byline">Creating Interactive Graph Visualizations</span> <em>Corey Lanum / <a target="_blank" href="http://watch.neo4j.org/video/64827612">Presentation (Video)</a></em></p> <p>The purpose of a visualization is to provide the user with a better understanding of the structure and relationships of the data. </p> <p><strong> Visualizing Nodes: Dos and Don'ts</strong></p> <ul class="style1"> <li>Use images and icons (as opposed to just circles with words in them)</li> <li>Use colors and/or gradients (gradients can be useful in expressing a numeric value, but don't get too fancy or combine colors and expect the user to know what you are expressing)</li> </ul> <p><strong>Visualizing Vertices: Dos and Don'ts</strong></p> <ul class="style1"> <li>Use colors to indicate categories of relationships</li> <li>Use width to indicate strength of a connection</li> </ul> <p><strong>Interactivity with the Graph: Dos and Don'ts</strong></p> <ul class="style1"> <li>It is important to allow the user to easily add and remove filters</li> <li>Provide an easy way to combine qualities and aggregate them under a node</li> <li>Animation: If you are going to animate, make sure you do it in a way that does not cause the user to lose track of individual nodes (Note, out of the box D3 does not smoothly animate)</li> <li>Layout: Don't force the user into the layout you think is best for the data. The user might be looking for something you were not looking for when you designed the application. Provide multiple layouts for the same data-set.</li> <li>Explore: One of the biggest advantages of graph DBs is the ability to discover correlations and causality you did not intend n finding. It is important you design visualizations with this in mind. This also means you don't need to show all the data you have at once, because it may not all be relevant. You can work around this by aggregating data under a node (think, "click to expand"). Aggregations can be done by grouping on common node properties, for example.</li> </ul> <p><strong>Visual Dangers</strong></p> <ul class="style1"> <li>Uniqueness: Not every relationship is useful (Why do all nodes tie back to one node? This can cause a lot of noise on a graph visualization and provides little to no value most of the time)</li> <li>Structural Rigidity: Don't assume that how you store your data is how users want to analyze/explore/think about the data.</li> <li>Don't overwhelm the user with data</li> <li>Don't try and be too clever. Make everything intuitive.</li> </ul> <p><strong>Common Mistakes</strong></p> <ul class="style1"> <li>Using 3D (causes occlusion problems, is hard to navigate, hard to print)</li> <li>Bad color choices</li> <li>No labels (What does this vertex actually mean?)</li> <li>No legend</li> <li>No tooltips on hover</li> <li>Complexity (Not providing filtering)</li> <li>No emphasis on important nodes</li> <li>Black backgrounds</li> <li>Bad navigation</li> </ul> <p><span class="byline">Natural Language Search with Neo4j</span> <em>Kenny Bastani / <a target="_blank" href="http://www.slideshare.net/KennyBastani/natural-language-search-using-neo4j">Presentation (Slides)</a> / <a target="_blank" href="https://github.com/kbastani">GitHub</a> / <a target="_blank" href="https://twitter.com/kennybastani">@KennyBastani</a></em></p> <p>In terms of a graph DB, you can search by simply traversing paths, matching all of the paths that are &#34;matches&#34; (based on any criteria), and returning the matches. Note, unlike a relational store, you are only paying the cost of the traversal (number of total nodes has little to no impact).</p> <p>It is also possible (and useful) to model time in a graph DB. Rather than modeling specific times and dates, Kenny explains that depending ont eh task at hand, it might be easier to model moments or snapshots in time. See <a href="http://gist.neo4j.org/?github-kbastani%2Fgists%2F%2Fmeta%2FTimeScaleEventMetaModel.adoc" target="_blank">this graph gist</a> for more details, but essentially modeling</p> <blockquote> T0 > T1 > T2 </blockquote> <p>...is helpful in the context of natural language search (or search in general) because you can model out things like:</p> <blockquote> At T0, Pam viewed Product X. At T1, Pam viewed Product Y. At T2, Pam purchased product X. </blockquote> <p>With a larger sample set, it is possible to make useful deductions on this data. For example, suggesting people who are viewing product Y to take a look at Product X. Kenny suggests storing all these path traversals in non-real time, storing them in a JSON-friendly store, and then at runtime, extracting the relevant traversals based on some action (e.g., <em>Viewing product Y</em>). Note, Extrapolating path traversals can be compute intensive, you may want to distribute &amp; map-reduce</p> http://www.Sanjay.io/2013/11/new-everything/http://www.sanjay.io/2013/11/new-everything/<p><span class="byline">New Domain, New Platform, New Host</a></p> <p>Well...I finally got around to tweaking my blog, and in theory that will mean I actually get to writing a bit more. This instance of my blog is running on <a href="https://github.com/Sandra/Sandra.Snow/" target="_blank">Sandra Snow</a>. It's basically a port of <a href="https://github.com/mojombo/jekyll" target="_blank">Jekyll</a> with some additional features, and of course running on .NET (<a href="https://github.com/NancyFx/Nancy" target="_blank">NancyFx</a>) rather than Ruby. The way both of these work is pretty much the same..</p> Fri, 15 Nov 2013 05:00:00 Z2013-11-15T05:00:00Z<p><span class="byline">New Domain, New Platform, New Host</a></p> <p>Well...I finally got around to tweaking my blog, and in theory that will mean I actually get to writing a bit more. This instance of my blog is running on <a href="https://github.com/Sandra/Sandra.Snow/" target="_blank">Sandra Snow</a>. It's basically a port of <a href="https://github.com/mojombo/jekyll" target="_blank">Jekyll</a> with some additional features, and of course running on .NET (<a href="https://github.com/NancyFx/Nancy" target="_blank">NancyFx</a>) rather than Ruby. The way both of these work is pretty much the same..</p> <!--excerpt--> <p>Create a mark-down files --> Run Sandra.Snow --> Get HTML pages --> Push to GitHub</p> <p>A mark-down file is really just a text file with some basic keywords (you can find more info on the specifics <a href="https://github.com/Sandra/Sandra.Snow/wiki/Markdown-File-Header" target="_blank">here</a>). Sandra.Snow basically reads these in, parses to appy the some rules (paging, etc.), and then spits out some pretty HTML which I push to GitHub (hello, entirely free hosting). This all means that when you hit my blog, you are hitting purely static HTML; zero database calls (well, unless GitHub is doing some).</p> <p>Thanks to <a href="https://twitter.com/philliphaydon" target="_blank">@PhillipHaydon</a> and <a href="https://twitter.com/jchannon" target="_blank">@jchannon</a> for their work on putting this together. </p>