There are a lot of methodologies available for development. I’ve done everything from Agile to Waterfall, either iterating as we go, or planning a really large project all the way through upfront. I’ve found pros and cons to all the different methodologies, but in the end, I prefer to take an iterative approach. The problem with iterating quickly, or as Mark Zuckerberg would say, “Move fast and break something”, is that you do, in fact, break things. In smaller codebases this might not be that big of a deal, but when dealing with larger scale production systems, breaking something is simply not tolerable. A good solution I’ve found is to implement a “Test First” development methodology where you building unit testing first, and then write the code to pass the tests.
Test First Development (or Test Driven Development) isn’t a new concept. In fact, the idea stems from the lack of traditional testing frameworks for most programming languages. When Node.js started, building in Unit Testing was one of the first things they did. This allowed developers to write small scripts that ran powerful unit tests on small sections of code. Unit tests are designed to test very small pieces (or units) of the code that accomplish small tasks. This is important because a particular function or module is often required to provide a standard response and format. If you change the underlying code, and it no longer behaves as intended, other functions and modules that rely on it may break, which could be catastrophic. Without being able to see which parts of the code are not behaving correctly makes debugging extremely difficult.
Abstraction in programming is a way to encapsulate functionality into reusable bits of code. A typical piece of functionality that would be abstracted is access to the persistent storage layer (databases, memcache, NoSQL, etc.). If every script you wrote had to connect to the database, check memcache, handle errors, etc., you’d be writing a lot of duplicate code. Plus each developer would need to have a complete understanding of the underlying data architecture. This is simply impractical.
To make life easier for developers and to protect the integrity of your datastore (you wouldn’t want a developer writing code that accidentally corrupted data), you would write a series of functions that handle the database interactions for you. You could then expose those functions to your developers so that they could easily grab data and not care about how the system handled the details behind the scenes. For example, you might create a Users model that accesses, creates, manipulates, and deletes users from your system. A developer writing logic would just need to call Users.get(user-id) which would retrieve the user information for user-id. Similarly, they could use Users.set() and Users.delete() to perform additional actions.
Putting it Together
By using a Test First methodology, all of the testing for your Users model is already written. You’d create tests that insert new users, manipulate users, and test the results that are returned. Assuming all these tests pass, then the expectation for other functions that use these should be met. If you change the underlying data model, you still need to make sure that it passes your tests. If it doesn’t pass your tests, then you need to review what other functions and modules may be calling these functions and update them. Otherwise, all of your tests will fail.
Integration testing is when your functions require underlying functions to work. This is typically what happens as you start writing tests for higher level components. Integration testing is important because it first assumes that all your low-level functions are working, then expects results from your higher level calls.
In the end, having a strong testing methodology in place is a must for growing codebases that need to be reliable. If you are writing production level code, having bugs, especially critical ones, are less than acceptable. Write good tests and test often.