Thoughts About Dependencies In Software World

Posted on December 20th, 2016 by ivan

Ideas

Everything in software is all about dependencies! Right this morning, this idea is becoming more and more clear in my head. So I decide to write down how this comes up in my mind and why I think it so. I will talk about how dependencies take effect in interfaces design, software architecture, deployment and productivity.

Dependencies

The definition of dependencies is quite simple: There are two things need to be done, but A has to be finished before B. It’s a little bit different in software since we involved time in dependencies, so the definition can be converted to “version X of A has to be finished before B” we could also saying “B depends on version X of A”. So as you see describing a dependency is still not very complicated but things could go wrong.

Problems In Dependencies Management

circular dependencies

Circular dependencies are when connecting all dependencies together it becomes a circle. The most simple example is A has to be finished before B and B also need to be finished before A. As you could see in above phase when this thing happens there is a conflict: A and B’s desire can not be fulfilled at the same time. In software when this thing happens software will most likely crash (yes there are some tricks for some specific languages like Python which can walk around it, but I highly recommend not to do the trick in most cases). In reality, things could be more complex, the circle can be huge when you aware things went wrong, and it could also be very hard to locate the real problem because there are many things involved.

There are two ways I recommend to resolve circular dependencies: one is instead of depending on each other, depend on a third party module. Take above example we could extract code from A and B out to C and let A and B both depends on C to resolve it. Another one is reorganizing the code put the dependencies either in A or B, because when circular dependencies happen usually means the code needs to be put together are separated into different places. About how to chose which method to use I have a very simple trick: if the code caused this problem is for specific business logic chose the second, or else if it also can be used elsewhere chose the first way.

Dynamic loading is also another way to fix the circular dependencies issue, but its use case is very limited, abuse it will lead to an even more complicated problem. For me, I will only use it for factories which could instance the underlying worker based on requires. Fox example in Java, Spring use reflection to instance service bean, in Python, you could use import system to dynamic instance service object and there are a lot many in other languages.

long dependencies chain

Long dependencies chain is that A depends on B, B depends on C and etc. When this chain grows longer and longer the robustness of software will be affected and also upgrading of the software will also be hard since you might encounter interface compatibility problem (remember the butterfly effect right :))

My suggestion for this problem is giving up on those third-party softwares which have long dependencies chain (usually the quality of those libraries are questionable) and write your own code. And if it’s your own code try to compress the chain by moving related code into a single place.

too many dependencies

Too many dependencies are the hardest in software, it’s more like a design issue at the beginning: When you need to do something different you always have to lookup in other libraries for help even if there may only be less than 100 hundred lines of code. The most famous example of this problem is “left pad” problem in Nodejs. I mean Nodejs definitely is a cool thing which really pushes the javascript world forward in a positive way, but the dependencies management sucks. Every time I initialize a javascript project it feels like I downloaded the whole javascript world to my laptop.

The thing for too many dependencies doesn’t actually have a way to fix, I think we can only avoid it by not flooding our project with too many dependencies, we have to keep an eye on our project to remove those unnecessary dependencies from time to time.

Dependencies In Interface Design

There is a misunderstanding about dependencies, people easily think dependencies are only those in configuration files (ant.xml, maven.xml in Java, requirements.txt in Python). No, it’s not. Dependencies are all about code splitting at the first beginning. This means when we design interfaces we need to make sure the cohesion of interfaces, interfaces which are similar should be put together instead of spreading everywhere causally. If you are confused about how to do it, exploring libraries which you use heavily, those usually can be treated as best practices. And remember don’t take only one library as the standard, check out more and compare them.

Dependencies In Software Architecture

For software architecture, the most important thing is the major framework your project depends on, for web backend project it’s the web framework(e.g.: Spring in Java, Django in Python, Rails in Ruby and etc), for frontend UI project (e.g.: React.js, Vue.js, bootstrap and etc). If you looked those project they either self-dependent or have very few dependencies. So from my point of view for those are heavily used in your code base best to depends on those which have simple dependencies.

And what’s more, avoiding use two similar things in one single project, predictable is one of the most important things for code readability, unnecessary confusion in code could make things unpredictable and failed eventually. If you have no choice then hide the underlying implementation by extract abstract interfaces suits your requirements.

Dependencies In Deployment

Yes, deployment is also involved with dependencies. Many companies have CI built to ensure code quality: usually, every time after the code is pushed to repository testing is triggered and if all test cases are finished without error code can be deployed to staging or production environment. So this is the simplest dependencies management in deployment and it just work. Think about if you do it manually how could you ensure this can happen every time in a correct way.

And the reality in deployment is more complicated according to the service size. Basically, it involves infrastructure(e.g.: network, hardware, backup, storage), service scaling and deployment, monitoring and log analysis and even more. Even more and more SaaS service and cloud service really simplified a lot of work compare to years ago, but it still quite challenging to integrate them together. Especially when micro-service is becoming popular nowadays dependencies management is also should be treated as the same important as them.

Automation and monitoring are the only things which I can think about to properly do the dependencies management for deployment. And there are actually tons of tools for it (Jenkins, Docker, Ansible, SaltStack and etc). When the complexity is out of the reach for humans we can only rely on those tools to behavior consistent and give us feedback when something goes wrong.

The End

Even I talked about dependencies in above sections, but still there are a lot of aspects I didn’t cover, but what I really want to urge people to do is looking into the things you are working on right now and try to figure out the relationships between them, even more, try to reorganize them if you find out it definitely an obstacle to your productivity and time will be saved for a better life-work balance.

Play With React&Webpack For Single Page App Development

Posted on June 26th, 2016 by ivan

Get Started

When I frist wrote this article I have been played with front-end for a couple of weeks, have to admit the javascript world changed a lot since last time I touched it about six years ago. In this post I will share what I have learned in working with React and webpack to build a single page app.

What Is Webpack & What I Use It For

From Webpack's website it claims itself as "module bundler" which actually took me a while to really understand what's the meaning behind. By using webpack we could make good use of ES6, sass, less, jade or anything with specific loaders to compile all those stuff into underlaying browser aware static assets. What's more, webpack does understand the dependencies, the compiled static assets can be split into smaller files with related content only, this is actually what I like the most. But Webpack is more than this, it supports plugins, so if you want to customize it: like compression, you could just apply a compression plugin to do it.

Webpack also provides tools for debugging: dev-tools and dev-server which make debugging a little bit easier. In short, I use webpack as a compiler for compilers, one tool to integrate all configuration for building static assets, a package tool but smart.

First Impression With React

React is "just the UI" framework, it encourages you to use it with other exist tools together to carry out M and C in classic "MVC" pattern, but react itself only focus on V. It has better performance because of virtual DOM and one way data flow to reduce the over complexity traditional bi-direction data bind involves. And another thing I like react is JSX, by using JSX the template and code can be separated more naturally which improves readability a lot. And with the life-cycle interfaces, React provides we could easily integrate with other non-react libraries.

Bird's-eye view For Tools I Use To Build A Single Page App

Webpack and React are like the core team members I use to build a single page app, but that's not enough, there are still some necessary components like local data storage, network interaction, UI Component libraries need to be handled properly. Below is a graph explains what are in the tool chain and how they communicate with each other.

As you can see above there are two types of arrows which marked as different color, the blue one is non-Ajax request and the orange one is Ajax request. All non-Ajax requests will return the same HTML page which is our application. In this single page it contains all the necessary javascript, css, images and etc. All the data requests will go through Ajax request which is represented by the orange arrow.

Look inside the application. As you can see Webpack is in the middle while the left side is our source code and the right side is the compiled static assets which are actually inside our application. Look at the left, the UI is built by React, on above redux will work as a storage for local state change, Axios is a network interaction component which the Ajax requests are sent here and the response is used to update the redux state object. Sass and Bootstrap are in charge of the appearance of the UI component. On the top is React Router, for myself react router's behavior is kind of similar to most backend framework's url mapper. As for an application it has many features, router makes it easy to split code by different business logic and provides a way to map the request(url) to the underlaying module.

As a backend engineer myself most of the time, I could easily bring a connection of this whole idea to backend web framework which react is like a template engine(not very accurate more like JSP/PHP actually), redux is a kind of database, axios is a RPC/Http client, React Router is a micro web framework and Webpack is a packaging tool(for example like maven). As we talked about backend and frontend so much before, there is actually no end in front of us ;)

For Development

In this section, I will talk about how the actual development life looks like for me, and you could find the related code in github https://github.com/ivannotes/django_project_template

Project Layout

▾ app/
  ▸ actions/
  ▸ apis/
  ▸ components/
  ▸ containers/
  ▸ reducers/
  ▸ stores/
  ▸ utils/
    App.jsx
▸ bundle/
▸ font/
▸ image/
▸ scss/

Above layout is the current project layout I use. The application is implemented in the app folder, bundle is the compiler output folder where the compiled static assets are located, scss folder is where I put the styles, font for font files, image for images. Ideas about the layout of app folder is borrowed from this article Presentational and Container Components components are those which could be reused, containers are more like controllers. For actions, reducers and stores are for redux, each folder is a represent of the related component in redux. apis folder is for ajax api, all the ajax interfaces are defined here, containers will call those api to read data from or write data to the backend.

Use Webpack To Compile Static Resouce

To use Webpack we only need a config file to setup everything. Below is a simple configure file I use and I will explain what each part of it is about in the following.

module.exports = {
    entry: {
        "app": "./app/App.jsx",
        "vendor": [
            "axios",
            "core-decorators",
            "react",
            "react-chartjs",
            "react-dom",
            "react-redux",
            "react-router",
            "react-router-redux",
            "react-tap-event-plugin",
            "redux"
        ],
    },
    context: static_project_path,
    output: {
        path: path.resolve(static_project_path, "bundle"),
        filename: "[name]-bundle.js",
        publicPath: "/static/bundle/",
    },
    module: {
        loaders: [
            {
                test: /\.jsx?$/,
                exclude: /(node_modules)/,
                loader: 'babel',
                query: {
                    // stage-1 for es-class-fields-and-static-properties
                    presets: ['es2015', 'react', 'stage-1'],
                    plugins: ['transform-decorators-legacy'] // for @ syntax
                }
            },
        ]
    },
    plugins: [
        new webpack.optimize.CommonsChunkPlugin(/* chunkName= */"vendor", /* filename= */"vendor-bundle.js"),
    ],
    resolve: {
        root: [
            path.resolve(static_project_path)
        ],
        extensions: ['', '.webpack.js', '.web.js', '.js', '.jsx']
    },
};

Context
Context is the basedir for entry points, I prefer to set it as the static project's root directory.

Entry & Output
Entry defines the entry points when webpack tries to compile your code which means webpack will start looking at the entry points defined to find the whole code base's dependencies. As you can see from the example above, there are two entry points defined: one is app another is vendor. The reason why vendor is separated from app is because vendor's code usually not changed or not that frequently changed as our application code, so split it into another file will help browser to cache vendor's code, but in order to split it we need an extra plugin to do that which will be explained later.

Loaders
Loaders give the possibility to preprocess files before it actually been required or imported. With this feature we could compile jsx into javascript, sass into css, coffee script into javascript and etc. In above example it defines how to handling jsx files, it uses babel loader to compile the jsx file with customized parameters in the query.

Plugins
Plugins usually related to the compiled file, in this example, it splits the common code(vendor's code) from application code, we could also use a plugin to compress javascript or css code.

Resolve
Resolve defines the mechanism how webpack find out the code when it required in the code base, above example defined the root and extensions which means the root path for webpack to looking for source code and the possible extensions. And notice that if you have code like

require("./somefile.ext")
with extension in it you must have an empty string in the extension definition.

Use React Router To Manage App Layout & Module

As I mentioned before React Router is very similar to the url mapper of backend web framework. Below code snippets can explain why I'm saying that.

<Router history={browserHistory}>
  <Route path="/" component={App}>
    <Route path="about" component={About}/>
    <Route path="users" component={Users}>
      <Route path="/user/:userId" component={User}/>
    </Route>
    <Route path="*" component={NoMatch}/>
  </Route>
</Router>

As you can see that each url can be mapping to underlaying React Component, And what's more the route can be nested into routes, which you can easily nest children into parent's node and this will help application layout a lot. Below is an example:

Users
<div>
    <div class="navigator-menu">
    </div>
    <div class="children">
        {children}
    </div>
</div>

User
<div>{user.name}</div>

So if I access /user/1232/ and assume the user's name is Tom, the page will be rendered as below:

<div>
    <div class="navigator-menu">
    </div>
    <div class="children">
        <div>Tom</div>
    </div>
</div>

See the children are filled into the parent's node. By using this feature we could define app's layout with menus and put all subpage as children then use different url patterns to switch from different sub pages.

Use Redux To Track State Change

As React see itself as an UI framework, we need extra help from other framework to keep track of state change. For example after user login, we need to store the user information somewhere for display, this is where redux come in for playing. Redux provide three components to track the state change of the application, they are action, reducer and store. Action is the data source for store, it defines what happens in the application and reducer responses to the action and decides the related state change in store, the store is where the state is stored. Another very important thing is that the store is basically a big data object.

Use Axios For Network Interaction

Axios is an http client library which based on Promise interface, it provides simple interfaces for network interaction. By simple configuration, you can start to play with it, and it also provides interceptors which you can inspect requests and responses which make some global process for requests and responses easier: for example global error handling.

Style

Use style is deadly simple, just add the css loader in webpack configure and then you can import it in your code. For example

import "./css/app.css"
, if you want to use sass then add another loader for scss and import it in your code. Below is an example for webpack to import bootstrap which is a famous style framework.

extra loaders

{
    test: /\.scss$/,
    loader: ExtractTextPlugin.extract(["css", "sass"])
},
{
    test: /\.css$/,
    loader: ExtractTextPlugin.extract("style-loader", "css-loader")
},
{
    test: /\.(woff|woff2|ttf|eot|svg|png|gif|ico|jpg)($|\?)/,
    loader: 'file-loader?name=' + '[name].[ext]'
}

With the above code and execute

npm install bootstrap
you could use the following code to import bootstrap into your code

import "./node_modules/bootstrap/dist/bootstrap.css";

But for myself I don't really like to import style sheet code in application, I like to split css code from application code, I use sass to give a help to organize css so I could easily split css from javascript. The idea is that I define an extra file import all the style sheet I need for the application then I configure another entry only for styles, remove all the imports for css in exist code and add

ExtractTextPlugin
to do the actually split. Below are the code snippets:

sass code:

@import '~bootstrap/dist/bootstrap.css';
@import './scss/app.scss';

Notice the

~
enable it to import from node_modules. After that, we will need to add the entry and plugins into webpack config file.

extra entry

"style": "./scss/style.scss",

extra plugin

new ExtractTextPlugin("[name].css")

Then you there will be extra style.css and style.js file generated, the style.js is a drawback of this solution, it only an almost empty javascript file you could just ignore it.

Debugging

The last part is debugging, which actually is where I spend most of our time at. For browser debugging source-map is good for connecting compiled code with source code, and webpack-dev-server is good for debugging with backend server.

devtool: "source-map",
devServer: {
    proxy: {
        '*': {
            target: devServer,
            bypass: function (req) {
                if (req.url.startsWith('/static/bundle')) {
                    return req.url;
                }
                return false;
            }
        }
    },
    host: '0.0.0.0',
    port: 9876
}

Above code snippets configured the source-map and webpack debug server, for dev server we could proxy all static request to our local environment and all non static request to backend dev server. To run the dev server I use the following command:

webpack-dev-server --port 9910 --inline --hot --content-base project/static/ --history-api-fallback

the --hot option enable the hot reload for static assets which make the dev process more efficient

After two weeks' playing on webpack and React, I feel like those two tools are amazing but there are also a lot of thing need to known before we can get them into our use. A lot of things still have space for improvements, for example webpack's documentation, but before everything is ready and prefect there are still tons of source for learning, there are tons of project on github related to those two, so download some famous open source project and try to learn from them helps me a lot, I borrowed a lot of things from sentry's source code in this post. Happy exploring!

More Articles