Package by type, -by layer, -by feature vs “Package by layered feature”

Florence, Italy

There are many ways to organize the classes in packages (folders, namespaces) in a software project. I am going to go over them from the most primitive to the most judicial and cover pros and cons.

Package by type

Because most of us software developers are self-taught, or we learn as we go, at the beginning we are too excited about having our code compile and run to think about highly abstract concepts such as packaging, architecture, design patterns, SOLID principles, etc.

But we have a natural tendency to be neat, and highly organized. When we start, we don’t even know that programming is art, we don’t know how much we don’t know, so we don’t know that packaging is art in and of itself. But since we are an organized bunch of folks, we start off with the good intention to group our classes in a very easy to navigate way. So what do we do?

We put all the classes of the same type in a folder named after the type. I’ve seen (and been involved with) projects where all the interfaces, because they are interfaces, are put together in an interfaces package. Then all the exceptions no matter where they are used, are in , you guessed it, exceptions package. The thinking goes, “I’m gonna have to work on an exception for this feature, let’s open the exceptions folder and find it there”.

Here is an example structure of a project neatly packaged in this way, bringing painful memories.

.activities
  - MainActivity.java
  - LoginActivity.java
  - RegisterActivity.java
  - SettingsActivity.java
  - BaseActivity.java
.fragments
  - LoginFragment.java
  - ForgottenPasswordFragment.java
  - RegisterFragment.java
  - ProfileSettingsFragment.java
  - AppSettingsFragment.java
.interfaces
  - ILoginListener.java // Don't get me started on Hungarian notation prefixing interfaces with "I"
  - IRegisterListener.java
  - IProductRepository.java
  - IUserRepository.java
.exceptions
  - FileLoadException.java
  - CameraException.java
  - TrackerException.java
.utils
  - NetworkManager.java
  - DatabaseManager.java
  - StringOperator.java
  - SharedPrefsAssistant.java // Oh yes, the "utils" package where all the gods, managers, assistants, operators, and their grandma live

So what happens if you work on the “Registration” feature?

• you need to expand all folders
• you need to find out which classes work with which classes from which packages in order to know where you need to make changes
• your Pull Request has merge conflicts with other co-workers, who worked on a totally different feature, but in the same (all) packages
• you keep scrolling up and down, reading all those names every time you need to switch the class you are working on
• virtually every package depends on virtually all packages, creating painful cyclic dependencies
• you can’t “ship” a module in isolation
• build times are painfully slow
• God forbid, you need to introduce abstractions, you have to touch all classes and packages everywhere
• when you open the project and are looking at the top-level package structure, you have no idea what this app does

Basically, if we used a real-world metaphor, and we organized our company this way, we would be doing the following

• all the chairs are piled up in one room
• all the desks are piled up in another room
• all the computers are in a third room
• all the documents in a fourth one
• all the humans in the conference room

Now, you need to perform a payroll operation. You enter the humans room, ask each person for their position until you find who’s the accountant. Take them by the hand, walk into the documents room, dive into the piles of paper, find out whatever contracts and documents you need, walk out, go into the computers room, find out which was theirs, get it, visit the chairs room, grab a chair, go where the desks are, and you get the job done.

You wanna fire an employee? Same procedure. You wanna outsource accounting? A massive mess occurs while every room is being turned upside down, looking for accounting items, documents, and people to get them out of there.

When a programmer has to live in such a code base, this literally translates to migraine, pain, unhappiness, lack of effectiveness, hating their job, hating their project, opening up Facebook, chatting with colleagues, hating the platform, and the company loses money. The developer suffers, the team suffers, everything goes down the drain.

Package by layer

After we’ve worked on a “package by type” project one too many times, we know better this time. We’ve read this or that example or a big company’s documentation and we’ve paid notice to sample projects. In the meantime we’ve also learned not to put “I” in front of interface names, to create Database or Network operation classes, and use more abstractions.

So the next step is to start thinking about architectural layers. We have the UI layer, the Network layer, the Database layer and the dreaded Model layer. The class organization patterns, sometimes erroneously referred to as architectures, such as MVC, MVP and their siblings, often go this way.

Here is how a package by layer project structure looks like:

.application
  - MyApplication.java
.database
  - UsersOperations.java
  - ProductsOperations.java
.network
  - UsersAPIService.java
  - ProductsAPIService.java
.models
  - User.java
  - Product.java
  - DBOperations.java // This is an interface, implemented by the classes in .database
  - APIService.java // This is another interface, outlining the interactions with the server, implemented by classes in .network. Included in this package, because it doesn't belong anywhere else?
  - TrackerException.java // Oops, now this exception is considered a model, because it is a class that doesn't fit elsewhere
.ui
  .activities
     - MainActivity.java
     - LoginActivity.java
     - RegisterActivity.java
     - ProductActivity.java
  .fragments
     - LoginFragment.java
     - ForgottenPasswordFragment.java
     - RegisterFragment.java
     - LogoutFragment.java
     - BuyProductFragment.java
     - ProductPageFragment.java
   .presenters
     - LoginPresenter.java
     - ForgottenPasswordPresenter.java
     - ProductPresenter.java
.util
  - SharedPreferencesManager.java
  - StringInterpolator.java
  - EventTracker.java
  - Logger.java

While somewhat of an improvement, this still bears most of the frustrated “package by type” downsides, namely:

• we still have to have all the packages expanded to work on a feature
• we still need to have memorized which classes work together and need each other
• there are still cyclic dependencies
• merge conflicts
• unable to ship modules as plugins
• all packages know about all packages
• very hard to decouple code, code to abstractions rather than concrete implementations
• the project structure still doesn’t say much about what kind of application we are working on
• build times are still slow
• And most importantly, gives us a false sense of security, as it looks more professional and thoughtful than just piling up classes in random folders for the lack of better alternative

Package by feature (a popular attempt)

Now, we’ve read a couple of books on architecture, we’ve read about Domain-Driven Design, we are familiar with the Clean Code family of architectures (Clean Architecture, VIPER, Hexagonal, Ports and Adapters etc) and we have learned how to

• model the domain of the application,
• decouple high-level policies from low-level concrete implementations,
• protect the domain entities from the implementation environment,
• invert dependencies by coding to abstractions, using repositories, decorators, facades, factories etc
• use the Open Closed principle, Interface segregation principle, Liskov Substitution principle etc

And we try our best to turn programming into a craft, and our project into a piece of art.

By this point we are also familiar with the software packaging principles and we deliberately think in this direction

(Cohesion:

• Reuse-release Equivalence Principle,
• Common-Reuse Principle
• Common-Closure Principle,

Coupling:

• Acyclic Dependencies Principle,
• Stable-Dependencies Principle,
• Stable-Abstractions Principle)

This is a very complex matter, and not exact science, and it takes many tries to get it right, if you can define right. It takes precision and judiciousness, but it is so rewarding.

Here are the benefits of combining Domain-Driven Design with the packaging principles:

• Every feature is in its own package and is built in isolation, without stepping on anyone’s toes. You only need to open one package and all the classes you need are there
• Features can be shipped like decoupled, modular plugins, like libraries
• If done correctly, there are no cyclic dependencies, and the build times should be faster
• The domain layer lives in its isolated universe, it doesn’t need to know which features are using it. It doesn’t need to know which platform it is running on. It can be converted into another language and put on a different platform
• Features can be deleted by a single right click on a single package
• When a change is requested, all the files that we need to change, are usually in the same folder — because according to one of the principles is that classes that should change for the same reason, should be grouped together.
• Adding new features is a breeze
• Refactoring or adding abstractions is a breeze
• Testing is easy
• Swapping one library or dependency for another is easy
• Makes everyone on the team think about modeling the entities, the layers, inverting dependencies, decoupling code, etc..

Here is a very simple example of a project packaged by feature, using Domain Driven Design:

.application
  - MyApplication.java
  - PreferenceManager.java
.domain
  .user
    - User.java
    - UserStorage.java // An interface, a contract listing all the storage needs we might have for a User, unaware of any particular implementation or a library
    - UserServer.java // An interface, a contract listing all the server API endpoints we might have for a User, unaware of any particular implementation or a library
  .product
     - Product.java
     - ProductStorage.java
     - ProductServer.java
     - ProductType.java // An enum
.features
  .login
    - LoginActivity.java
    - LoginFragment.java
    - LoginNetworkOperation.java
    - LoginInteractor.java
    - LoginPresenter.java
    - LoginMVPContract.java
  .forgottenpassword
    - ForgottenPasswordFragment.java
    - ForgottenPasswordNetworkOperation.java
    - ForgottenPasswordListener.java
  .product
    .create
      - CreateProductUseCase.java
      - CreateProductActivity.java
      - CreateProductNetworkOperation.java
      - CreateProductDBOperation.java
      - CreateProductListener.java
      - CreateProductException.java
    .buy
      - BuyProductUseCase.java
      - BuyProductActivity.java
      - BuyProductNetworkOperation.java
      - BuyProductDBOperation.java
      - BuyProductListener.java
      - BuyProductException.java
.network
  .user
    - LoginUserRetrofitImpl.java
    - CreateUserRetrofitImpl.java
    - LogoutUserRetrofitImpl.java
  .product
    - CreateProductRetrofitImpl.java
    - BuyProductRetrofitImpl.java
  - RetrofitNetworkManager.java // A particular library which all of the classes in the package use to implement interfaces from the domain layer
.storage
  .user
    - UserRepositoryGreenDaoImpl.java
  .product
    - ProductRepositoryGreenDaoImpl.java
  - GreenDaoStorage.java // A particular storage library which all of the classes in the package use to implement interfaces from the domain layer
.tracker
  - EventTracker.java
  - TrackingException.java
  - UserLoggedInEvent.java
  - ProductBoughtEvent.java
.logger
  - Logger.java
  - LogException.java

• All the tracker code is in its own package
• If we wanted to stop using Green Dao as a DB and we wanted to swap it for an in memory cache, we just wipe all the GreenDaoImpl classes, and we create InMemoryCacheImpl classes that conform to the same storage contract protocols, without even touching the domain layer
• All the logger code lives in its own package.
• If we wanted to remove the “Forgotten Password” feature, we just delete the package, no one else is bothered
• The domain layer doesn’t know anything about the features, they depend on it, it doesn’t depend on them. There are no cyclic dependencies between the domain and the plugin features. This will ideally lead to better build times.
• Features can be turned on/off or shipped as separate modules

Now, this example isn’t very complete, it doesn’t come from a real project, there are still things to be messed around with, some classes missing, but I am trying to paint the picture of where this is going, and I think it lives up to this task. Also, not all packaging principles are adhered to fully, and the example is too simple to showcase where they shine. Not to mention, I am still not completely fluent in this.

Package by layered feature

I don’t think this term exists, but this is a twist of mine on the package by feature methodology, which introduces packaging by layers within feature packages, and I think it gets the best from both worlds — a feature package, where you can clearly outline the network layer, the view layer, and the storage layer, which is something that “package-by-layer” people use to bash “package-by-feature” projects, where it is sometimes hard for them to find where all the networking logic lives, if they needed to change all the networking classes.

This methodology resembles matrix organizational structures, where there are legal departments, accounting departments, and marketing departments for instance, but there are cross-department committees and teams, where a line is cut through all departments and includes one person from each department in a cross-functional team.

So let’s just take a look at a subset of the project structure:

.features
  .user
    .login
      .view
        - LoginActivity.java
        - LoginFragment.java // Implements LoginMVPContract.View
        - LoginPresenter.java // Implements LoginMVPContract.Presenter
.network
        - LoginRetrofitImpl.java // Implements LoginNetworkOperation
        - LoginServerResponse.java 
        - LoginListener.java
     - LoginInteractor.java (or LoginUseCase.java) - coordinates the login flow, gets data from the .view, fetches user from the network, and calls the storage classes to save the user in local DB, then pings the UI again.
     - LoginMVPContract.java // Idealistic presentation layer contract, doesn't care that the view is a Fragment or how the presenter is implemented.
     - LoginNetworkOperation.java // Idealistic network contract, doesn't care that the particular implementation is by Retrofit, in the .network subpackage
.product
    .create
      .view
        - CreateProductActivity.java // Implements CreateProductMVPContract.View
        - CreateProductPresenter.java // Implements CreateProductMVPContract.Presenter
      .network
        - CreateProductRetrofitImpl.java // Implements CreateNetworkOperation
        - CreateProductServerResponse.java
        - ProductNetworkEntityToProductMapper.java // Maps the network representation of the product to our idealistic Product.java entity in the domain layer which doesn't know how the server guys model their products, and doesn't care about networking libraries.
      .storage
         - CreateProductGreenDaoRepositoryImpl.java // Implements CreateProductStorage.java
      - CreateProductInteractor.java
      - CreateProductStorage.java // Idealistic contract interface, could be implemented in many ways
      - CreateProductNetworkOperation.java // Idealistic contract interface, could be implemented in many ways
      - CreateProductMVPContract.java // Idealistic contract interface, could be implemented in many ways
// These contracts are perfect for testing, they can be easily mocked, making testing a breeze.

So there you have it, an attempt at “Package by layered features” which is in my opinion, the most evolved way to create a robust project. Of course, this should only be applied to massive enterprise level projects, because everywhere else, it would be vastly over-engineered.

Each feature is now its own standalone project, packaged by layer, having the contracts in the top level directory, and particular implementations in the subdirectories.

Projects that benefit from this structure typically:

• Have all the layers (view, network, storage, file system, sensors) — if your project lacks some or most of those, then this methodology might be an overkill
• Need to be tested
• Need to be modular, or be able to add features as plug ins
• At any point, each feature can turn into a self-contained library
• Could benefit from shorter build times
• Are worked on by developers who are aware of architectural principles and can precisely choose how to name, abstract, and group classes. This can turn into a massive mess or lose its purpose if not implemented correctly
• Have a lot of stuff going on, so not having to expand all the packages to work on a feature is of big help to developers
• Overlapping features can all be subpackages of a bigger package that contains the shared code
• Could be transferred to another platform at some point. A well designed “Domain-driven design” domain can be copied and pasted into another platform, translated to their syntax and reused. Same goes for a big part of the code too
• Need to experiment with different network, file system or storage implementations or libraries, and could benefit from being able to swap one implementation for another with absolute ease. If we wanted to replace the network implementation, we only needed to replace the `NetworkImpl` classes in the `.network` subpackages without changing anything else
• Need to experiment with view layer organizations. All presenters can be swapped with viewmodels and nothing else has to change!

I’ve tried my best to keep this article as short as possible.