There are quite a few good books about Software Architecture. So I think the literature defines Software Architecture well. But what you are really asking about is where the Architect fits into the Software Development Team. With the change over from Agile and Scrum approaches to Software Development the architecture and requirements seems to be left out of account in most of the agile process models and so it seems that the architect role and requirements role have been eclipsed and we seem to think that architecture and requirements are no longer needed and systems can accrete, but of course this leads to Technological Debt. So what I would like to address is the agile at scale where the project is big enough to need the differentiation of requirements and architecture to produce a coherent product.
Now the problem seen with Architecture and Requirements is that they produce documents that are not executable, and thus get out of date, and are not kept up throughout development, and thus are seen as waste from a lean perspective. And as you say all developers have a view of the whole system they are working on in their heads, so why do we need architecture? I am going to try to address that question.
We should recognize that Design itself is applicable at all levels of System and Software development. Thus developers who do not consider themselves or are not considered “architects” are doing design when ever they create something new or different in software. We have software patterns, but the combination and adoption of the patterns is still a design task, even if we do not have to make up new patterns. Design is ubiquitous in development and no one has exclusive access to design within software development, and especially if we adopt agile methods, we should realize that everyone owns the design, and autocratic approaches that give design rights to one developer over others may be expedient but in the long run is harmful, because everyone on the team needs to share the architectural design of the product they are developing together. We know how to share implementation, but we are not as good at group self-organization of the architecture. But the self-organization of the team should reflect the self-organization of the architecture.
I would like to mention that I have just done my dissertation on Emergent Design and what I am going to say is based on that research See http://about.me/emergentdesign
The problem is with agile (scrum) type methods is that although they espouse self-organization of the team there is no real theory of self-organization upon which they are based. Therefore I would like to offer a model of self-organization upon which I will base my remarks.
See also http://www.mediafire.com/?y122wwv7d89qp5u
I have constructed a model of software process based on this model in my book Wild Software Meta-systems. http://works.bepress.com/kent_palmer/
So given that context let me begin by saying that self-organization can be seen in the nature of knots that are organized against themselves by their own self-interference. So we have a very precise model of self-organization in knot theory. Knot theory is a new discipline, and probably one if the youngest of the mathematical disciplines. The best work on this is that of Louis Kauffman http://www.math.uic.edu/~kauffman/ whose work I follow. Particularly because he is interested and takes seriously the work of G. Spencer Brown in Laws of Form. http://www.lawsofform.org/
Self-organization is related to self-production which is Autopoiesis, and we are talking about teams that produce these complex systems in most cases so that means that we have reflexive autopoietic systems exhibiting this self-organization. Self-organization of the team, and the organization of the product developed by the team are two different but deeply related matters. What we need to understand better is how the alleopoietic organization of the product of the team reflects the self-organization of the team.
In other words we are arguing that requirements and design are intrinsic to the self-organization of the team, and that this is reflected in the requirements and architectural design of the product produced by the team. So let us consider what requirements and how they relate to functions. Requirements have to be developed in relation to a functional model of the system. This is because without the functional model you would not know whether the requirements set is complete, consistent, or clear or balanced, or fully describes the intended system. Requirements are ideally Godellian statements of the hypothetical emergent properties of the system that is desired. I like the statement that I read somewhere that the requirements carry the customer value of the system being developed, and allow us to keep our focus on that during the development process. But Requirements are un-ordered, they are axiomatic statements.
The physical and functional architecture are related to the Agent and Functional viewpoints on the realtime system which are partially ordered. But those partially ordered viewpoints ultimately have to yield embodiment in spacetime as eventdata computation that is fully ordered. Thus the relation of requirements, design, and implementation are determined by what Klir in Architecture of Systems Problem Solving, the methodological distinctions with respect to the ordering of variables in the system.
At the highest level we have unordered requirements, and then at the next level we have partially ordered functions and agents, then at the next leve we have either partial order with distance or linear order without distance which are duals of each other. Finally we have full order with distance which is the condition that is necessary for computation and full implementation. So when we say organized, there are different levels of order that we may be alluding to.
Now what is fascinating is that once we realize that the methodological distinctions orders determine the level of organization which is present in the system being developed and that there is a duality in this lattice between linear order without distance or partial order with distance. These two duals have the kind of duality that exists in the minimal methods. The minimal methods are the information order that is necessary to capture the relations between viewpoints. Since there are four viewpoints on a realtime system there are six different transformations between these viewpoints. Interestingly these more or less correspond to what exists as methods in UML and SysML. Functions were taboo in the original UML specification but this minimal method has been added back in with SysML. The only difference between object oriented and functional dataflow minimal methods is whether you are looking at data from the point of view of function or function from the point of view of data.
Now once we understand the minimal methods as bridges between the viewpoints on a real time design we can start thinking about this in both allopoietic and autopoietic ways in order to relate the self-organization of the team and the alleopoietic organization of the architecture of the system.
My own research led me to create the Integral Software Engineering Methodology (ISEM) which was a domain specific language to give a model based description of software architectural designs. Recently I gave a paper on how this might be updated to work at the Systems Design level based on the research in my dissertation. This paper can be seen at http://kentpalmer.name which is my resume page. But at the bottom you can see my CSER paper on the changes that I have made to my design language to make it easier to do Systems Design.
First I advocate using Domain Specific Languages to express design. You can see why in my critique of SysML: http://holonomic.net/sml01a03.pdf.
The key idea that instead of diagrammatic visualization that we get in UML and SysML we should express designs in text like we express code. This way as text we can keep it like the code and even put it within the code as comments. It could act as constraints on the code structure and thus be part of the execution strategy when we build the code or even be the basis of code generation. The problem with UML and SysML is that they are semantically weak being composed of only relations.
So let us go back to the question at hand, which is how the self-organization of the team is related to the organization of the structure of the software. Architecture can be seen as a protocol for organizing team coordination over the structure of the product being built. In other words the team provides a reflexive social meta-system within which individual team members function together and coordinate their development work. But part of the information they have to convey to each other and preserve is about the internal structure of the application that they are building. Requirements and Architectural Design is part of the protocol for communicating about the structure of the program that is based on the self-organization of the team. In other words the team is producing an intersubjective synthesis. The production process is structured and that allows the product to be structured at the macro level. But this means there must be a protocol which will allow the team to develop different parts of the system without working against each other or stepping on each others work, such that the whole system works together when integrated, verified and validated.
If we see requirements and architectural design as this communication protocol that holds the macro synthesis together then it is possible to see how architecture and requirements are nothing different from the value stream of the team by which the emergent properties sought are projected into Being. If one is doing software within some predetermined framework then the framework is giving the order to the whole application that the various team members are contributing to. But if the application has a unique structure that needs more than the framework then we will need an architecture that is designed to give a macro structure to coordinate work.
As a person who does not understand narrative very well intrinsically I have become fascinated with the topic through a personal deficiency.
One thing that I can understand is that narratives are inversions of maps. In other words narratives are to time what maps are to space.
So as we can say the map is not the territory we can also say that the narrative is not the temporality.
We can consider the fractal nature of time. http://www.if-online.org/Fractal%20Time%20pdf%20file.pdf It is flowing at all scales.
An excellent example of a narrative map is at http://xkcd.com/657/large/
A good example of a map narrative is “Everything Sings: Maps for a Narrative Atlas” by Denis Wood http://www.sigliopress.com/books/atlas.htm
The relation of Map to Narrative is a spacetime interval. This means there is a phase space between time and space within the interval, such that from some points of view the map is bigger and the narrative smaller, and from other points of view the narrative is bigger and the map smaller.
So you would think that if I can understand maps and diagrams I would by a simple transform be able to understand narratives. But it really does not work that way. I can understand narratives of works of art that I do structural analysis of, but I cannot invent a narrative myself. I manage to tell stories in daily life but I cannot make up one. I found this out when I tried to write an Epic See http://archonic.net/epic/index.htm
My problem is that to me all possible paths are the same, and I don’t know how to choose between them. This is nihilistic of course. But what are we to do . . .
No one for a long time has written a book that encapsulates the worldview. Dreyfus gives some examples from history like the Anead, lliad/Odyssey, Dante’s inferno, Brothers Karamazov, or Moby Dick. He has a new book about that called the Shining Ones.
I have a fundamental handicap that I cannot understand how to produce a narrative, even though the narative is merely the inverse of the map. So although I know the structure of the Worldview, I have not figured out how to turn that in to a Novel. I started writing and Epic Poem of the last adventure of Odysseus (http://archonic.net/epic/index.htm) in order to attempt to produce a story that reflected my knowledge. I agree with Heidegger that Poetry and Thinking are intertwined but allow you to get at different kinds of depth. I tend to attempt to write poetry rather than narratives myself. But unless the worldivew is posited as a narrative then it is not possible to absorb it easily for others entrapped in that worldview.
So lets invent something. First I would like to posit the premise that Male and Female differ in that Males are oriented toward Space and Females toward time. And so the spacetime interval appears in human relations between genders. Now smith said that genders are a kind of a kind, but if you think about it we can see that in terms of category theory that the genders are actually modifications. Thus there are arrows, functors, natural transformations, modifications and fluctuations as the first five levels of N-Category theory. So one thing to think about is how the various relations between people, say a couple or a family or a group are conditioned by the various N-Categories with gender falling at the modification level.
Now the way that genders relate to the worldivew is in terms of the initiation ceremonies. It turns out that Apollo and Artimis are the masters of initiation for the two sexes, with boys becoming wolves and girls becoming bears. Now it turn out that boys and girls experience the meta-levels of Being in different orders in the initiation ceremonies, but the initiation ceremonies are complementary. See my Jung reading group presentation at http://archonic.net/jung_alchemy_presentation_2000_03_09.PDF and http://archonic.net/ago00v00.pdf.
It turns out that these sequences in the initiation ceremonies are the same as the sequences through life that the different sexes follow.
The story of Ariadne maps out the female initiation ceremony. Perseus maps out the male initiation ceremony more or less. Goux has described Oedipus as a failed hero who has aborted the initiation process.
So I think there is a story in that. But how to turn it into a narrative I am not sure. I can map it, but cannot temporize it properly into a narrative.
This is actually a complicated question to answer because so many things are being assumed in the question and the first anonymous response which I take to be the questioners further explanation. However, the question itself does beg for an answer. Is Monotheism a precursor for Modern Science and our progress in it which has changed the world so much? This core question is pretty significant if it were true, that monotheism was a prerequisite. Personally I don’t think this is so. And my evidence for that is that China discovered everything that was available in the Renaissance a thousand years earlier and then forgot them again and again. The real key to progress (or regress, as it may be) is not forgetting and adding various inventions on to each other to produce new inventions. This is something that Indo-Europeans managed to do, and I think the concept of Being, unique to them, has something to do with their success in this regard. I think Being is a precursor. In other words I don’t think it is the semitic contribution of monotheism that is as important. However, there is no doubt that it played a role by supplying through synthesis the idea of the Supreme Being when Monotheism and Ontology are conflated. But these are the contributions of the two nomadic tribes in the Middle East. But there are also the contributions of Egypt and Mesopotamia, which we are just now starting to appreciate, because it had been lost for so long. All of these factors contributed to the foundations of Science through the production of a meta-worldview, which we now live in as the world wide dominant worldview which is quickly supplanting all the others. It could be argued that Science would not be what it is if it did not have to struggle against Christianity. And it is certainly true that Reason defines itself against the superstition of religion. But the fact is that Reason existed in Plato’s academy which lasted for a long time, and through Euclid’s elements persisted to become the core of the educational system in the West. In Euclid’s elements we get a condensation of the knowledge of Being and its relation to rationality. But it took a long time for rationality to show the kind of results in science that would allow it to pull free of religious superstition. But on the other hand, rationality itself fully developed in a Polytheistic society. One could argue that Christianity merely delayed the rise of Scientific culture though the intensification of superstition after the Roman era. The seeds were sown by the take over of the world by Alexander which seeded Athenian Greek culture across the known world. But Romans were not interested in theory, but only practical applications, and when Rome fell apart after adopting Christianity, then the dark ages and general european collapse ensued until the Renaissance when there was a rediscovery of the Classical past in which Reason was given pride of place within a polytheistic context. Thus one could argue that Reason is the child of Polytheism and that it was only with the re-introduction of polytheistic elements during the Renaissance that that Reason could gain a foothold again, as something other than a support to religious superstition. However, it is way more complicated than that, because polytheistic societies like Greece and Egypt, were in fact really monotheistic at core, which we see in Plato and Aristotle where they talk about God and the gods. It was assumed in those societies that there was a Godhead, and that all the gods were merely manifestations of that godhead. So it is not even true that Egypt and Greece were completely polytheistic because they understood well that there was one god behind the various manifestations of that reality behind the appearances of the gods. And so we could make a case for the fact that Greece and Egypt were monotheistic in a sense similar to that of the Semites up until the second destruction of the temple. After that second destruction Judaism purified its Monotheism and that is when radical monotheism with one god and no others entered the scene. But before then it was always assumed that beyond polytheism there was a deeper monotheism. Same is true in Hinduism. Polytheism is itself not pure, but a kind of Monotheism that allows differentiation into gods who manifest the powers of the hidden single Godhead (Brahman) behind all the phenomena of its manifestation in the world as separate powers and attributes which explains why the world is such a mess, where evil came from etc.
So I think this is a question that could have several competing answers and it would be hard to disentangle what the true precedents are for science. Monotheism in this deeper sense can be one of them, but it is not clearly the case.
I wrote the first draft of my dissertation Emergent Design at http://emergentdesign.net in six weeks. Then it took ten or eleven months to rework it so that the english was acceptable with my wife as my editor. We argued over every sentence. After I was done I did a diff between the original draft and the final passed dissertation, there were a few words that went untouched, like I noticed that the phrase “In other words, . . . ” on one page remained untouched throughout the editing process. Not sure how those words were spared.
I am a prolific writer. Prolific does not necessarily mean a good writer, as my wife keeps reminding me. But there is a certain point when one begins to be able to just set down and start writing. Taking that stream of consciousness and making it into something worth reading is then a lot of work, which I do not do except for things that are published. So I have a lot of very doubtful writings on my web sites and what is available (over 10,000 pages) is just the tip of the iceburg of what exists off line or in manuscript form. Writing is my way of thinking things through. I learned to write working papers in college, and never stopped. I also learned to do diagrams as the basis of my thinking. When you combine that with a lot of reading, almost ten years sitting in the British Museum reading everything in Philosophy and myriad other subjects I could get my hands on, and almost every book that existed was available there, then eventually you think you might have something to say.
I know, I know that is all illusory. But it keeps me going, now writing answers on Quora, as an advertisement of my more serious works available on my web sites. I tried going to conferences but that did not work, so here I am slaving away answering questions on Quora in hopes that someone out there will find Special Systems and Emergent Meta-systems interesting.
At a certain point I became fluent in my writing. And after that all I had to do was sit down and start writing. As I write I discover things that I did not know before, and that gives me the elevated interest and really fascination or thrill of discovery that leads me to read on in the myriad subjects of my interest, which is fairly wide. I will never forget after I received approval for my Ph.D. in England which took about nine years, all my Advisor had to say was, now you have a general education. In other words he was saying, you are just beginning, and you now have the basis for going further and doing your own research on your own. One thing leads to another and after a while you have read a lot of pages and written a lot of pages in response to what you have read. And years have gone by. Some parts of life you missed out on, myriad parties I suppose. But on the other hand when all is said and done, a little knowledge goes a long way. My favorite book along these lines as I have mentioned previously is Knowledge Painfully Acquired by Lo Chin Shun.
There is someone who spent their whole life thinking very deeply, who came to a conclusion after myriad avenues explored, and set his conclusions down succinctly as a guide to others. Being prolific but not being a very good writer has many disadvantages, one of which is that your work does not get read even if it deserves to be read. So my advice is to learn your lessons well, and then strive to become prolific after becoming proficient. Those who become prolific prior to becoming proficient are doomed to fail even if their ideas are very good.
However, failure in terms of getting attention should not be counted as overall failure. The actual reward is in the discovery process itself. The thrill of the chase in to a territory never visited by anyone else, not mapped out, and with no beaten tracks. It may be rare in the earth, but in the landscape of ideas there are still many continents left to discover, and there the local inhabitants do not mind.