Figure 1. Advantage of WANG-based software development.
Copyright © 1996, All Rights Reserved.
No parts of this document should be reproduced without prior consent from
the author.
John C. Wang is a graduate student at the Intelligent Robotics Lab at the National Taiwan University, Taiwan. He can be reached from the WWW at http://www.csie.ntu.edu.tw/~jcwang/index.cgi or by email at jcwang@csie.ntu.edu.tw .
The Internet is most often touted as the new medium that will connect people together. One immediate consequence is that more people will soon have the luxury of working together at distant locations and, with the realization of the new technology, be productive in ways which we may not have expected. Create! is an initiative to provide an Internet-based environment supporting the kind of collaborative project development that could be becoming the popular style of collaboration over the Internet. This report focuses on our particular approach that help people form collaborative Wide Area Networked Groups to reach the right teammates on the Internet, and discusses our findings, trends, opportunities and problems encountered in the process.
In the spring of 1996, a major software company plays catch up in a software race. As a result thousands of developers were put into a division in an attempt to capture the market by sheer force. Its major competitor, trying hard to save market leadership, also ballooned to a size of 1,200 employees within a year of market presence [6]. Industry analysts doubt if the software fever will go on forever. It will be interesting to see how the companies reposition their huge work force when the race stops.
Perhaps the companies could have fought an easier war with possibly greater results had they utilized access to enthusiastic software professionals from all over the world. Properly arranged, collaborative teams and the company can work together in the time of crisis, without an entire fleet left standing and become a burden long after the event. Obviously this time the technology was not there to make the kind of collaboration work.
Imagine one day in the year of 2001, a software developer ponders under an aged oak in the center of a great pastureland. With his laptop computer and a satellite link, he is a happy and creative freelance software developer who actually gets to choose his boss and coworkers over the network. Five years ago only the supremely venerable senior architects perhaps get the freedom to work away from the company. Now proven collaboration policies free professionals from bindings to companies. Alliance formed with the technology yield better results since the participants chose to work for their global reputations and are personally responsible for staying genuinely focused and dedicated.
Figure 1. Advantage of WANG-based software development.
The advance in collaboration developed in this report is what we termed Wide Area Network Groups, or simply WANGs (Figure 1). To qualify as a WANG, a team must form through an unobstructed, fast and far reaching public communication channel, so that anyone who is capable has a chance to join. Participants advertise through the channel about their expertise, areas of interest, and past work history. They also communicate about project details, decide on design issues, and manage progress over the very same medium. We should expect to see WANGs, with its high degree of flexibility and dedication within, outperform rigidly bound and inadequately informed corporate task forces. The first part of this report gives supporting evidence in the context of organizational and communication studies.
Developing software using WANGs, as this report demonstrates, comes with a choice of a wide variety of professional backgrounds, and can actually pinpoint those who are interested in the nature of the project. A case study at the end of the report illustrates a new and attractive software development process we have discovered while working with WANGs. This observation gives us hints that there might be fundamental differences and advantages in working in WANGs compared to working with traditional software teams.
The central technology of our presentation was developed in the Ideae Create! Project (web site at http://ideae.csie.ntu.edu.tw/Create/). We propose that the WWW provides a great gathering place for WANGs to advertise themselves and for software systems acquirers to reach the proper teams. Here we try to establish a clear record on the development of Ideae Create! so that a foundation exists for interested readers to reproduce and experiment with WANGs in various professional sectors.
Perhaps the year 2001 will be too much for us to foretell. For now we take a moment to roll back in time and examine the fundamental factors that is leading to collaboration. From there we retrace the steps of Ideae Create! and take a closer look at how a site can be engineered to provide support for wide area collaboration among WANGs.
Collaboration involves the coordination of workers through some communication channel. As the scope of communication becomes more far reaching, interactions among workers and organizational structure become increasingly complex and sophisticated. New organiztions are formed naturally as people learn to take advantage of improved communication methods.
Besides communication, one other significant impact on collaboration is the innovation in organizational structures. Three types of changes (Figure 2) have been observed and summarized below.
| Traditional Structure | New Structure | |
|---|---|---|
| Traditional Medium | huge, monolithic companies and corporations | strategic business networks, self-managing teams |
| New Medium | home offices, virtual companies | wide area collaboration over WANGs |
Since the 1980's, various organizational models have been pursued and experimented with, resulting in promising concepts including strategic networks [5] and self-managing teams [4]. Each of these directions grants the worker a higher degree of autonomy and responsibility. For industries where technology is a primary factor to competitiveness, workers are reported to make more contribution with an increased degree of participation.
Even without the use of improved communication hardware, these new organizations are already achieving better collaboration by flattening hierarchies to shorten the length of communication transactions, and by promoting task-oriented teams to encourage communication among workers.
Arguably, a silver bullet mechanism for organizational structural change is still non-existent. Much the same is the point we are at with collaboration technology. For organizations that are too risky to undergo dramatic structural change, communication technology is now available to support some form of improved collaboration.
Using virtually the same technology that has made the World Wide Web popular, corporations are deploying intranets today and moving employees towards telecommuting, effectively forming virtual companies [7]. These mechanisms help save a great deal of operation costs, and the proper use information supports employees working over long distances.
Combining the advantage of both advancements in organizational structure and the utilization of modern communication media, collaboration over the wide area has drawn much interest and shown promising signs.
The application of the World Wide Web is drifting towards encouraging wide area, open interchange of information to promote collaboration. The Open Meeting project [3], whose initial goal is to "[make] use of computer networks to coordinate policy planning and actions across traditional organizational boundaries," is part of the U.S. government's effort to arrive at a more participatory alternative to the traditional process of governmental policy making. Continuing the work in the field of CSCW (Computer Supported Cooperative Work), The BSCW Shared Workspace System [1] uses the WWW to complement present CSCW solutions that "facilitate information exchange, [yet] provide little support for information sharing."
In this report, we present WANGs as an advancement towards ideal collaboration.
Even though much can be said about real collaboration needs the organizations are craving for, little research has been conducted to identify the procedures needed to render the results. In the WWW today, a very small number of sites are starting to formulate the procedures needed to mass produce collaborative services. A couple brave and note worthy pioneers are listed below.
Ideae Create! is a project that fits into the big picture as another WWW-based service that is open to the public. We provide a directory service that enables developers to reach each other for forming WANGs, and allows companies with software needs to be able to gather a great team for solving their problems.
The following sections are aimed at satisfying the reader's curiosity and also for providing a set of how-to's for those who are interested in reproducing our results. From as early as the initial concept that eventually led to the full-blown WWW site hosting two hundred software developers, there are critical ideas and concepts to observe and lessons to learn.
In April of 1996, Create! was conceived as a possible solution for arranging matches between software project developers and clients. This was first proposed to solve the problem that there are always outstanding programming jobs that cannot find programmers other than the handful who are always busy. Conversely, there are always skilled programmers who are getting a slack and would not mind taking on an extra part-time job, but usually can't find one right off. Very soon our naive goal was leading us to solutions which we later identify as the Wide Area Networked Groups.
Developing software using an open environment like the WANGs has advantages over traditional methods. Some of which are listed in Figure 3.
| Traditional Software Development | WANG-based Software Development | |
|---|---|---|
| Access to Developers | Limited by the size of the company (usually under 100). | Interested and qualified engineers from all over the world. |
| Communication of Concepts | Scattered notes from formal, costly and infrequent meetings. | Automatically archived from ad hoc electronic messages. |
| Project Plans | Printed documents usually filed away and forgotten. | Serve as a schedule for electronic contacts and file transfer. |
| Prototyping Cycle | Ineffective due to the cost of installation and delivery. | Distributed by email or file transfer. Fast and low cost. |
In May, a prototype of Create! was developed on the WWW, using a CGI-like but more efficient variation called the ISAPI (Internet Server Application Programming Interface). The original prototype ran on the Microsoft Internet Information Server 1.0, and included features that span a chain of HTML pages. To our surprise, the prototype system was so fragile that it regularly causes system crash after executing just a couple of the common functions. A few days after the opening of the test site, many of the early users simply gave up trying.
In the original Create! design, each member of Create! is first registered online as either a Create! Developer or a Create! Client. Once approved, a Create! Developer has access to a list of recently posted software development requests. If the Developer chooses to take on a job, she registers the choice through the check boxes on the web page. A simple agreement is presented, including the duration and approximate payment upon completion. These terms and conditions were specified online by the Create! Client who posted the request.
The design of the original prototype was an overkill, causing unnecessary complications when there are only a handful of users around in the system. Operations that span more than a couple web pages (including those for posting and taking on software projects) become a nuisance especially when combined with frequent network delays. On June 25, a simpler service concept was proposed and was presented to the public. The new site is formally named the "Create! Athlete."
Create! Athlete is a WWW site consisting mostly of dynamic pages generated from the database on the fly. The programming interface used is the Internet Data Connector (IDC) provided with the Microsoft Internet Information Server. IDC was one of the first to efficiently generate web pages using the Structured Query Language (SQL). An immediate advantage for choosing a well-defined database language is the higher stability of the system. Actually, Create! has never crashed due to a programming fault since.
In Athlete, the entire Create! site is redesigned to fit a simpler database report scheme: one database operation per page, and hence an overall more responsive user interface. Even though the initial server load is relatively light (roughly 3000 pages accessed per day), by using an efficient data scheme we have ensured that the service will scale with future growth in the number of users.
Also simplified at this stage is the process of joining and browsing Create!. The registration process is now down to just one email request or one online registration form. From the front page of Create!, fewer than three mouse clicks is all it takes to see detailed information on any Developer.
Faster and convenient information access, coupled with greater system reliability, Create! is finally ready to serve the users with the right information interface.
Within weeks after the opening of Create!, applications from Developers poured in and demanded immediate processing, or the piled up data will soon become a permanent burden. In addition, decisions must be made about how the processed data will be stored away so they can be easily retrieved later on. Is full text search feasible? How costly is it to implement a particular data structure? How effective will it be? Finally, it will not be long before we get piles of server logs to process. Is there an advanced data mining tool appropriate for analysis? How do we detect user behavior and needs from the logs collected?
Create! maintains lists of Developers who register either online or through email, as well as lists of projects that are posted by Clients also through email. Since most of these information come in free form, they demand line-by-line natural language process. Fixed form, multiple choice style surveys might have helped, but at least initially we dare not to limit the possibilities by restraining the presentation. We are not aware of any natural language processing tool that is capable of comprehending the full spectrum of language usage in a professional resume, much less would we expect them to be available for handling Chinese.
Our scheme for attributing a description to a Developer was actually simple and extensible: we construct an ontological structure based on each description about a Developer. A hierarchy on expertise and experiences forms incrementally as each entry is added. For example, when a Developer wishes to register an expertise in Device Driver Programming, we add the node \Expertise\Hardware-related programming\Device drivers into the ontological tree; if the attribute given was System Analyst at Xyz Computers, we add \Experience\Software systems analysis/design\Professional software company\Xyz Computers. Each node has a parent node. For the two examples above, their parents are \Expertise\Hardware-related programming and \Experience\Software systems analysis/design\Professional software company, respectively. The backslash is used to separate depths in the ontology.
The translation from free form text into the ontological representation is reasonably simple for human processing, and the value of the data stored in this form can be tremendous. First, it would be very easy to implement a service so Clients can walk up and down the tree and explore the collection of Developers sharing similar traits. Also it makes possible for multilingual translations of stored information with relatively low effort (translating just the nodes instead of full texts). Right now there are under four hundred nodes in the entire tree used at Create!. This is a significant reduction from the thousands of lines we would otherwise have to store.
By October, after in operation for four months, Create! has collected some nine megabytes in server logs and half a megabyte in Developer information. From the very beginning of the project, user trails have been primary indicators of directions for improvement. The easiest way for us to perform analysis on the collected data is to utilize existing data processing tools, rather than to take time out of our already crammed maintenance schedule to design custom-made tools.
Since the server software we used automatically collects log data into relational databases, it was particularly easy for us to generate daily reports on access statistics using standard relational database languages. Each day the operator would use predefined database queries to determine the number of times each page entry has been accessed in the past twenty-four hours. Another query is defined to generate daily access totals so we can plot long term trends with standard graphical tools. Since most of the data used at Create! are kept in standard formats, it is quite simple to move the them back and forth among various tools for extensive analysis. Had we chosen to code our own report generating tools using custom data formats, we might be spending most of our valuable time wrestling with the bits and digits.
One way to make a WWW-based service more attractive than the traditional way of doing things is to provide creative and novel services that are truly useful. Out of the millions of new things we can do, one immediate question to ask is which few of these would really help users improve their online experience and also raise the quality of the development process?
If Create! is to be a project aimed at supporting real people with real needs, it would be very important that the users are assured that they can trust the people they work with on Create!. Otherwise it would be hard to convince users to give Create! enough respect to count on it for developing critical software projects. And in that case we will not have the chance to show that there might be better alternatives to greatly enhance quality and productivity in software development.
Create! provides information for software developers and clients to find each other on the Internet. How is the inclusion of WWW services making it different from a bulletin board at a labor service bureau? What aspects of the new medium can we take advantage of to make it superior to the traditional way? When we asked the users what new features they want, in most case they are unable to state a direct answer. It is much easier to make a potential improvement and then see if the users are interested in using it regularly. A few methods listed below could shed light on some of the useful improvements we can make.
Communication for collaboration on the WWW takes a lot more than just web pages and relational databases. Today's communication media include news groups and emails on the low end of the bandwidth spectrum, electronic white boards, Internet phones and video conferences on the high end, not to mention the traditional telephony and facsimile we had for decades. In addition, new mobile computing devices are being deployed as we speak. What made these ideas popular? What can we learn from them?
In fact, in the Ideae Developer's Suite project already underway, a number of the features mentioned above have been included to improve the support for different aspects of software development. More about the project is discussed later in this report as directions for future research.
Discovering the trends before users become aware of their needs is to some degree enough to keep the service comfortable and attractive to use. However, tools accessible to the common user today are so primitive that they usually do not offer hints or explanations beyond simple frequency counts. The study of data mining techniques promise to yield solutions, and it remains to be seen how much the tools can do in discovering traces that the common user can't already guess.
As we learned in the process of talking to the users on Create!, if there is one thing the will make WANG-based collaboration work, it is the mutual trust among the team members. The Internet is an open, unrestricted space exactly praised for these properties, but the very same virtues make it impossible to track the real identity and physical location of another party on the Internet. The underdeveloped level of trust among the Create! Developers and Clients is by far the biggest problem that inhibits the growth of Create! and affects the quality and seriousness of the participants.
An ideal credit system would give complete history of previous projects a particular Create! Developer involved in and comments his partners and Create! Client had about him. It would include both records of accomplishments, as well as cases of failures. It is unreasonable to demand that a person never makes a mistake, and wrong doings described in detail can help a future project partner decide whether the mistake would affect the performance in a new collaboration.
A proper credit system can also help people set themselves apart from others. A Developer with an outstanding credit history is in a position to take on projects of his choice at a salary better than the average Developer. Anyone with an empty history is treated as a beginner and would only be handed the most fundamental and least risky components. This avoids the problem that a Developer committing serious faults would disguise as a newcomer. Since he will be handed the least risky and most well controlled task, it is likely he won't be able to make a serious mistake again. Of course, people with an unfavorable history would have to settle for a lower wage.
With a good credit system installed, a software team would have a clear choice of Developers from the collection of backgrounds. A team composition would depend on the allotted funds and the required skill levels.
In order to appreciate the potential difficulties Create! Developers might face, I decided to be personally involved in a software project of the WANG style. The project initiated in October of 1996, and, as it happened, involves a Create! Client (a pharmaceutical trading company) and two Create! Developers (I and another programmer), over a distance spanning the entire island of Taiwan (approximately three hundred miles). The goal of the project is to construct a multi-user, security enforced, GUI-based inventory and accounting system for managing the sales of medicine. The amount of transactions handled by the system is estimated at over four million U.S. dollars annually.
To demonstrate the effectiveness of using WANGs, internally we set a goal to have the project completed within one month, instead of the three months allowed by the Client. There will be a period of two months to make up for unforeseen problems. I was responsible as a contact point and a project manager to gather a dream team for the task, to make sure that everything goes by the ideals of WANGs, and that nothing ever gets out of control. The project is internally named Project Danben from the name of the Client's company.
The process which we later termed Distributed Incremental Model is put to test in this project. In the beginning, an email was sent to dozens of Create! Developers who are listed at Create! specializing in developing management information systems. The email asks for authors of existing inventory and accounting systems who are willing to modify their system to suit our needs. They were asked to provide a clear, detailed plan describing exactly how they will go about revising their product to accommodate the new specification. Of the five Developers who responded, two had the ideal systems and plans for the project, and were selected for further review.
With in a week, a Developer was selected who happens to live three hundred miles away from both the Client and me. Our communication was mainly carried over the Internet using the WWW and emails. A plan was set, according to the Developer's own proposed schedule, ending on November 24. In the meantime the Developer and I would hold revision cycles at a period of five days. For each cycle, an incomplete, but executable version of the final system is delivered to me over the Internet, and I am responsible to review the version and give requests for changes on the following day. The changes requested would be incorporated in the next delivered version. The reports and requests written in emails are there to stay as implicit agreements that are easy to refer to in future email discussions. As the process goes on, the product is incrementally revised to fit the final requirements of the project. Should anything go wrong in the process, we would always have a working system that satisfies a useful portion of the final system. We would then have an option other than building everything over again from the ground up.
To summarize, the benefits of the Distributed Incremental Model are in two aspects. One is the flattening of risks throughout the entire development process. Fatal failures are detected at an earlier stage, making it less costly to abort a failing project. The lowered risk makes it safer to collaborate with unacquainted partners. The other aspect of the benefits is the required software reuse. Since systems are started from an already built system instead of from scratch, it is very likely that a large portion of the new system is the result of reusing the old. It is also very likely that the new system is constructed to accommodate later reuse. Not many traditional software methodologies can offer the equivalent features. The benefits mainly came with the adoption of a flexible, fast, and low cost communication medium.
Everything went fine in Project Danben until half way into the project with two weeks remaining. The Developer working with me moved from a software company to another, causing him to put aside our project to settle down in the new environment. He did not want to give up the project, but he really needed to catch up with the new job. This report offers no step-by-step methodology to resolve the kind of situation. What exactly is needed to make sure the project carries on is pretty much an art at this point.
After a serious delay in the fourth revision cycle, I talked on the phone with the Developer to clear up questions we could not resolve using email. Before the conversation I have already contacted the runner-up Developer who earlier also offered an excellent project schedule. He is up and ready to take over the project and deliver his system given a month's time. After the conversation, it was decided that there will be no replacement in project personnel, and an week will be added to the original plan to make the situation more controllable. Working with the Developer and understanding his problems, instead of setting strict rules and penalizing him for delays, has saved the project from the cost of a complete overhaul. However, it is not to say that strict rules and penalties are never options. There is too much human factor in the situation and, like most management problems, we do not yet know an universal mechanism to determine a solution.
This case gives hints on problems and solutions buried deep inside software development over WANGs. A lot of the problems remain to be seen, and there is definitely plenty of room for future investigation.
This experiment has made clear many issues that were far from being obvious in the beginning. A few of the delightful results include:
When people talk about collaboration on the Internet today, most would mention communicating using the email, joining electronic forums for discussions, or carrying out teleconferencing over private corporate networks. There is rarely a systematic approach to define or evaluate the role these communication services play in collaborations. If a particular method works, there is little we can say about why it works. If a successful service is discovered, there is not a guideline we can follow to determine how it fits in with other existing services.
In an attempt to answer these questions, the ongoing series of Ideae Collaboratics projects target specifically at understanding how information tools and services help people conduct collaboration. In the Ideae Create! Report we hope to have illustrated a model that brings benefit to software developers through collaborative Wide Area Networked Groups. We wish to have demonstrated an implementation and a set of accomplishments for future implementations to take note of and to compare against. For the next step, we should focus on understanding how best to coordinate among various existing and future information devices to yield maximum benefit for supporting collaboration.
Collaboration today is a wild beast refusing to be tamed. According to a recent study, "in pure statistical terms, failures [in collaborative ventures] is almost as prevalent as success." [2] As a computer scientist, perhaps you and I can help improve the situation by looking ahead and develop concepts and systems that can help lay the infrastructure beneath the eventual collaborative applications. This may be one of the many opportunities we should feel humble about our limited knowledge and learn to develop on top of establishments from fields including organizational management and social psychology. Perhaps with our concerted effort, there will be a day when collaborating is as common as breathing air. But for now we are definitely missing most of the critical pieces.
Last Update: December 4, 1996
Created: October 9, 1996